Sep 07 10:18:26.438405 ip-10-0-141-95 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 07 10:18:26.438570 ip-10-0-141-95 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 07 10:18:26.438681 ip-10-0-141-95 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 07 10:18:26.439011 ip-10-0-141-95 systemd[1]: Failed to start Kubernetes Kubelet. Sep 07 10:18:36.522130 ip-10-0-141-95 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 07 10:18:36.522147 ip-10-0-141-95 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2cddbcd1412b4bd99e5a6fe2b7b9ac03 -- Sep 07 10:20:32.040976 ip-10-0-141-95 systemd[1]: Starting Kubernetes Kubelet... Sep 07 10:20:32.468374 ip-10-0-141-95 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:20:32.468374 ip-10-0-141-95 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 07 10:20:32.468374 ip-10-0-141-95 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:20:32.468374 ip-10-0-141-95 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 07 10:20:32.468374 ip-10-0-141-95 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 07 10:20:32.469106 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.468371 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 07 10:20:32.471778 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471753 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.471778 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471776 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.471778 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471781 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471784 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471787 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471791 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471794 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471797 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471800 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471805 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471807 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471810 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471812 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471815 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471817 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471821 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471824 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471826 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471829 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471831 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471834 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471837 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.471881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471840 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471842 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471845 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471847 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471850 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471859 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471861 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471864 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471866 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471869 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471871 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471874 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471877 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471880 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471884 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471887 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471891 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471893 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471896 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471899 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.472366 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471901 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471904 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471906 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471909 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471911 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471914 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471916 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471919 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471922 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471924 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471927 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471929 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471932 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471934 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471937 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471940 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471942 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471945 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471949 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471954 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.472886 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471956 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471959 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471961 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471964 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471967 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471969 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471972 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471975 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471978 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471981 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471987 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471991 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471994 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471996 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.471999 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472001 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472003 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472006 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472008 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.473404 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472011 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472013 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472016 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472018 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472020 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472023 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472025 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472027 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472031 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472034 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472037 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472039 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472044 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472047 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472050 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472053 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472056 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472058 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472824 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.473988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472831 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472835 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472845 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472850 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472854 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472856 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472859 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472862 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472865 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472868 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472870 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472873 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472875 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472878 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472880 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472883 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472885 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472888 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472891 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472894 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.474440 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472897 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472900 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472903 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472905 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472908 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472910 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472913 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472915 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472918 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472920 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472923 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472925 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472928 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472930 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472933 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472936 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472938 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472941 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472944 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472946 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.474958 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472949 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472953 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472958 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472961 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472965 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472970 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472973 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472975 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472978 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472980 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472983 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472985 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472988 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472991 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472994 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472997 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.472999 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473002 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473004 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.475451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473006 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473009 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473011 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473014 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473016 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473019 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473022 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473024 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473027 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473029 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473032 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473035 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473037 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473040 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473042 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473045 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473048 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473050 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473055 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473057 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473060 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.475932 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473062 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473065 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473067 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473069 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473072 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473075 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473078 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473080 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473083 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473085 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473088 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473090 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473093 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473097 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473100 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473102 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473106 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473108 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473524 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.476437 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473530 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473533 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473535 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473539 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473541 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473544 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473546 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473549 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473551 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473554 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473558 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473561 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473564 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473566 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473569 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473572 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473574 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473577 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473597 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473600 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.476936 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473603 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473606 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473609 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473612 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473615 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473619 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473623 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473626 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473629 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473632 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473635 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473638 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473641 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473643 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473646 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473649 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473652 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473654 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473657 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473660 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.477421 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473663 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473666 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473672 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473675 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473679 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473683 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473686 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473690 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473693 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473696 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473699 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473702 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473705 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473707 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473710 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473713 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473715 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473718 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473720 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.477954 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473723 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473726 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473728 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473731 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473733 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473736 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473738 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473741 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473743 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473747 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473752 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473755 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473758 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473761 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473763 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473767 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473770 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473772 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473777 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.478458 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473780 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473782 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473785 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473787 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473790 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473792 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473795 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473797 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473800 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473803 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473805 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473808 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473810 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473813 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473815 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473818 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473820 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473823 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473826 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.473828 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.478948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473912 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473947 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473953 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473958 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473963 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473967 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473971 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473975 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473979 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473982 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473986 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473991 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473994 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.473997 2577 flags.go:64] FLAG: --cgroup-root="" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474000 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474003 2577 flags.go:64] FLAG: --client-ca-file="" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474007 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474010 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474014 2577 flags.go:64] FLAG: --cluster-domain="" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474017 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474020 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474024 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474027 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474031 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 07 10:20:32.479434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474034 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474037 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474041 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474044 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474047 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474050 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474053 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474056 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474060 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474063 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474066 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474069 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474072 2577 flags.go:64] FLAG: --enable-server="true" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474075 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474080 2577 flags.go:64] FLAG: --event-burst="100" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474083 2577 flags.go:64] FLAG: --event-qps="50" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474087 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474090 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474092 2577 flags.go:64] FLAG: --eviction-hard="" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474098 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474101 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474103 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474107 2577 flags.go:64] FLAG: --eviction-soft="" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474110 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474113 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 07 10:20:32.480031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474116 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474119 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474121 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474124 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474127 2577 flags.go:64] FLAG: --feature-gates="" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474131 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474134 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474137 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474140 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474143 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474146 2577 flags.go:64] FLAG: --help="false" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474148 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474152 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474155 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474158 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474161 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474164 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474167 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474170 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474173 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474176 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474179 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474182 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474185 2577 flags.go:64] FLAG: --kube-reserved="" Sep 07 10:20:32.480663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474187 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474190 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474194 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474197 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474199 2577 flags.go:64] FLAG: --lock-file="" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474202 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474205 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474208 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474213 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474216 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474219 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474221 2577 flags.go:64] FLAG: --logging-format="text" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474224 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474228 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474230 2577 flags.go:64] FLAG: --manifest-url="" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474233 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474238 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474241 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474245 2577 flags.go:64] FLAG: --max-pods="110" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474248 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474251 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474254 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474257 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474260 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474263 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 07 10:20:32.481273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474266 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474273 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474276 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474279 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474282 2577 flags.go:64] FLAG: --pod-cidr="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474285 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474292 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474294 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474297 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474302 2577 flags.go:64] FLAG: --port="10250" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474305 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474308 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c80fe02dc0ffd636" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474311 2577 flags.go:64] FLAG: --qos-reserved="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474314 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474317 2577 flags.go:64] FLAG: --register-node="true" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474319 2577 flags.go:64] FLAG: --register-with-taints="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474323 2577 flags.go:64] FLAG: --registry-burst="10" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474326 2577 flags.go:64] FLAG: --registry-qps="5" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474329 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474332 2577 flags.go:64] FLAG: --reserved-memory="" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474335 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474338 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474341 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474344 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474346 2577 flags.go:64] FLAG: --runonce="false" Sep 07 10:20:32.482002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474349 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474352 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474355 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474358 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474361 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474364 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474367 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474370 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474372 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474375 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474378 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474381 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474384 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474387 2577 flags.go:64] FLAG: --system-cgroups="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474390 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474395 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474399 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474402 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474406 2577 flags.go:64] FLAG: --tls-min-version="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474409 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474412 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474415 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474418 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474420 2577 flags.go:64] FLAG: --v="2" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474424 2577 flags.go:64] FLAG: --version="false" Sep 07 10:20:32.482905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474428 2577 flags.go:64] FLAG: --vmodule="" Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474432 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.474435 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474557 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474561 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474564 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474567 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474570 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474573 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474575 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474577 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474596 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474599 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474602 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474604 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474609 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474612 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474615 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474618 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.483980 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474621 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474624 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474627 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474630 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474634 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474637 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474639 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474642 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474644 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474647 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474649 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474652 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474654 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474657 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474660 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474662 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474664 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474667 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474669 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474672 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.484765 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474674 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474677 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474679 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474681 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474685 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474688 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474690 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474693 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474695 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474698 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474700 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474703 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474705 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474708 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474710 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474713 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474717 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474719 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474722 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474724 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.485261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474727 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474730 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474732 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474735 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474739 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474742 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474745 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474747 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474750 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474752 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474755 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474757 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474760 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474762 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474765 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474767 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474771 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474773 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474776 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.485950 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474778 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474781 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474783 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474786 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474788 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474790 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474793 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474795 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474798 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474802 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474806 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474809 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474812 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474814 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474817 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474820 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474822 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474825 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474828 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.486881 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474831 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474833 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474836 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474839 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.474842 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.475546 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.484624 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.484654 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.485542 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.485556 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485678 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485688 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485693 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485698 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485702 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.487510 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485707 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485711 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485716 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485721 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485725 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485729 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485734 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485738 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485742 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485747 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485751 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485755 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485759 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485763 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485767 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485771 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485775 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485779 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485783 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485787 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.488013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485791 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485796 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485800 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485806 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485816 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485820 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485824 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485829 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485835 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485842 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485847 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485851 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485856 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485860 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485865 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485870 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485874 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485878 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485882 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485886 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.488852 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485891 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485895 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485899 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485903 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485907 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485911 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485915 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485919 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485923 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485928 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485932 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485936 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485941 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485945 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485950 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485955 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485959 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485964 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485967 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.489561 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485972 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485978 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485984 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485990 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485994 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.485999 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486003 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486007 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486011 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486015 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486019 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486023 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486028 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486032 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486037 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486041 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486045 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486049 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486053 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.490110 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486057 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486061 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486066 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486069 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486073 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486078 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486085 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486091 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486096 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486102 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486106 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486110 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486115 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486119 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486124 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.490701 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486128 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.486136 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486361 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486371 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486376 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486381 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486385 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486390 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486394 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486398 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486402 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486406 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486410 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486414 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486420 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 07 10:20:32.491076 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486424 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486429 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486433 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486437 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486441 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486445 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486450 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486454 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486458 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486462 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486466 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486472 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486476 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486480 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486486 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486490 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486495 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486499 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486503 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 07 10:20:32.491515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486507 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486511 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486515 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486519 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486522 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486526 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486530 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486534 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486538 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486542 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486546 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486551 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486555 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486560 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486564 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486568 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486574 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486600 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486606 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486611 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 07 10:20:32.492007 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486615 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486619 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486623 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486626 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486631 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486635 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486639 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486642 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486646 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486651 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486655 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486659 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486663 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486667 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486670 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486674 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486678 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486682 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486686 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486690 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 07 10:20:32.492515 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486695 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486699 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486705 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486710 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486715 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486720 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486725 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486730 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486735 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486739 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486744 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486748 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486753 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486757 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486761 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486766 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486773 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486780 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486784 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 07 10:20:32.493039 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486789 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486793 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486798 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486802 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486806 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486811 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486815 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:32.486820 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.486828 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.487028 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.490057 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.491229 2577 server.go:1019] "Starting client certificate rotation" Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.491744 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 10:20:32.493734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.491785 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 07 10:20:32.498717 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.498693 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 10:20:32.503638 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.503611 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 07 10:20:32.518967 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.518939 2577 log.go:25] "Validated CRI v1 runtime API" Sep 07 10:20:32.523469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.523446 2577 log.go:25] "Validated CRI v1 image API" Sep 07 10:20:32.524555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.524523 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 07 10:20:32.524707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.524669 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 10:20:32.532497 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.532464 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9b4ede89-645e-42cf-9378-a2dd839eecb9:/dev/nvme0n1p3 d82223fd-a292-44ff-aec0-278da0dbf2fd:/dev/nvme0n1p4] Sep 07 10:20:32.532497 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.532493 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 07 10:20:32.537010 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.536890 2577 manager.go:216] Machine: {Timestamp:2026-09-07 10:20:32.536368583 +0000 UTC m=+0.394057889 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088225 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8b0bd0364267fec3314c2d8bcd83 SystemUUID:ec2b8b0b-d036-4267-fec3-314c2d8bcd83 BootID:2cddbcd1-412b-4bd9-9e5a-6fe2b7b9ac03 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dd:52:e8:bc:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dd:52:e8:bc:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:56:77:35:2b:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 07 10:20:32.537010 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.537001 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 07 10:20:32.537137 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.537094 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 07 10:20:32.540479 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.540448 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 07 10:20:32.540652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.540484 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-95.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 07 10:20:32.541969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.541955 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 07 10:20:32.542007 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.541971 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 07 10:20:32.542007 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.541979 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 10:20:32.542954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.542942 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 07 10:20:32.544753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.544739 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 07 10:20:32.548237 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548218 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pkn2k" Sep 07 10:20:32.548763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548748 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 07 10:20:32.548932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548921 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 07 10:20:32.548968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548955 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 07 10:20:32.548997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548969 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 07 10:20:32.548997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.548988 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 07 10:20:32.549050 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.549000 2577 kubelet.go:398] "Adding apiserver pod source" Sep 07 10:20:32.549050 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.549010 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 07 10:20:32.549453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.549440 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 10:20:32.549488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.549463 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 07 10:20:32.552145 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.552123 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 07 10:20:32.556942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.556701 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 07 10:20:32.556942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.556734 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pkn2k" Sep 07 10:20:32.556942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.556759 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 07 10:20:32.557489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.557471 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 07 10:20:32.557660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.557554 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 07 10:20:32.557718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.557664 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 07 10:20:32.557950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.557823 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 07 10:20:32.558169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558153 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558172 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558196 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558204 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558217 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558232 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 07 10:20:32.558250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558246 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 07 10:20:32.558461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.558255 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 07 10:20:32.559185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.559168 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 07 10:20:32.559635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.559617 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 07 10:20:32.560895 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.560875 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-95.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 07 10:20:32.561122 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.561108 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-95.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 07 10:20:32.561167 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.561122 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 07 10:20:32.563771 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.563757 2577 server.go:1268] "Started kubelet" Sep 07 10:20:32.563854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.563826 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 07 10:20:32.563904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.563853 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 07 10:20:32.563948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.563930 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 07 10:20:32.564734 ip-10-0-141-95 systemd[1]: Started Kubernetes Kubelet. Sep 07 10:20:32.565138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.565122 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 07 10:20:32.565571 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.565550 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 07 10:20:32.570039 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.570006 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 07 10:20:32.571351 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.571333 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 07 10:20:32.571351 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.571346 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 07 10:20:32.571469 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.571442 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.572141 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572122 2577 factory.go:55] Registering systemd factory Sep 07 10:20:32.572141 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572143 2577 factory.go:223] Registration of the systemd container factory successfully Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572315 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572312 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572342 2577 factory.go:153] Registering CRI-O factory Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572354 2577 factory.go:223] Registration of the crio container factory successfully Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572330 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 07 10:20:32.572450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572445 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 07 10:20:32.572780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572466 2577 factory.go:103] Registering Raw factory Sep 07 10:20:32.572780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572487 2577 manager.go:1196] Started watching for new ooms in manager Sep 07 10:20:32.572780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572501 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 07 10:20:32.572780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.572511 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 07 10:20:32.573369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.573032 2577 manager.go:349] Starting recovery of all containers Sep 07 10:20:32.577120 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.577039 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:32.581925 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.581840 2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-95.ec2.internal\" not found" node="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.585531 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.585508 2577 manager.go:354] Recovery completed Sep 07 10:20:32.590989 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.590975 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.593084 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593063 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.593159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593102 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.593159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593114 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.593798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593782 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 07 10:20:32.593798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593795 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 07 10:20:32.593910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593813 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 07 10:20:32.593910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.593796 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.594674 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594657 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.594734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594715 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.594734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594730 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.594900 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594889 2577 policy_none.go:49] "None policy: Start" Sep 07 10:20:32.594932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594905 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 07 10:20:32.594932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.594915 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 07 10:20:32.595752 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.595740 2577 policy_none.go:47] "Start" Sep 07 10:20:32.640332 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640308 2577 manager.go:337] "Starting Device Plugin manager" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.640357 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640372 2577 server.go:87] "Starting device plugin registration server" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640700 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640712 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640824 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640939 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.640949 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.641433 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 07 10:20:32.647996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.641496 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.714880 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.714836 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 07 10:20:32.716190 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.716171 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 07 10:20:32.716266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.716233 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 07 10:20:32.716266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.716259 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 07 10:20:32.716337 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.716307 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 07 10:20:32.718608 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.718518 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:32.741578 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.741536 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.742678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.742644 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.742678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.742678 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.742826 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.742689 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.742826 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.742714 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.751552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.751529 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.751698 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.751561 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-95.ec2.internal\": node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.768790 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.768761 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.817111 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.817053 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal"] Sep 07 10:20:32.817207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.817158 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.818232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.818218 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.818295 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.818247 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.818295 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.818257 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.819549 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.819536 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.819688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.819661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.819736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.819692 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.820383 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820364 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.820472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820398 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.820472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820414 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.820472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820399 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.820579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820490 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.820579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.820504 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.821532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.821515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.821617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.821544 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 07 10:20:32.822320 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.822304 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientMemory" Sep 07 10:20:32.822367 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.822333 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasNoDiskPressure" Sep 07 10:20:32.822367 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.822342 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeHasSufficientPID" Sep 07 10:20:32.846493 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.846466 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-95.ec2.internal\" not found" node="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.850961 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.850941 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-95.ec2.internal\" not found" node="ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.868881 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.868842 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.874209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.874189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.874298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.874239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.874298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.874257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17f670fc079e23e62690da1f2f92b580-config\") pod \"kube-apiserver-proxy-ip-10-0-141-95.ec2.internal\" (UID: \"17f670fc079e23e62690da1f2f92b580\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.969962 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:32.969896 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:32.975317 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17f670fc079e23e62690da1f2f92b580-config\") pod \"kube-apiserver-proxy-ip-10-0-141-95.ec2.internal\" (UID: \"17f670fc079e23e62690da1f2f92b580\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.975374 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.975374 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.975374 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.975477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17f670fc079e23e62690da1f2f92b580-config\") pod \"kube-apiserver-proxy-ip-10-0-141-95.ec2.internal\" (UID: \"17f670fc079e23e62690da1f2f92b580\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:32.975477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:32.975400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d6c692e6d81acebe6a01d57406b92c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal\" (UID: \"2d6c692e6d81acebe6a01d57406b92c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:33.070828 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.070785 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.149341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.149304 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:33.153146 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.152756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:33.171657 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.171450 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.272138 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.272027 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.372540 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.372498 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.473057 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.473009 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.491382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.491347 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 07 10:20:33.491537 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.491503 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 10:20:33.491601 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.491529 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 07 10:20:33.562016 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.561903 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:15:33 +0000 UTC" deadline="2028-02-12 01:22:29.212256454 +0000 UTC" Sep 07 10:20:33.562016 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.561958 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12543h1m55.650302347s" Sep 07 10:20:33.573386 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.573342 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.673802 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.673756 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.774730 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.774682 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.788751 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:33.788705 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f670fc079e23e62690da1f2f92b580.slice/crio-1ce9fbd0d219afa0cf3181304268376d65191960aea99a6dae73af69398b1c6e WatchSource:0}: Error finding container 1ce9fbd0d219afa0cf3181304268376d65191960aea99a6dae73af69398b1c6e: Status 404 returned error can't find the container with id 1ce9fbd0d219afa0cf3181304268376d65191960aea99a6dae73af69398b1c6e Sep 07 10:20:33.789148 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:33.789124 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6c692e6d81acebe6a01d57406b92c1.slice/crio-c08530e0d5a2bdf56f1e97f3fdab51f409b6b8cdfbe2d0ec0257ce7341054f98 WatchSource:0}: Error finding container c08530e0d5a2bdf56f1e97f3fdab51f409b6b8cdfbe2d0ec0257ce7341054f98: Status 404 returned error can't find the container with id c08530e0d5a2bdf56f1e97f3fdab51f409b6b8cdfbe2d0ec0257ce7341054f98 Sep 07 10:20:33.794308 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.794289 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:20:33.875493 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.875347 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:33.965667 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:33.965636 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:33.976180 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:33.976147 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:34.076270 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.076231 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-95.ec2.internal\" not found" Sep 07 10:20:34.108217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.108188 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:34.175278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.175175 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" Sep 07 10:20:34.187182 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.187146 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 10:20:34.188053 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.188038 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" Sep 07 10:20:34.202921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.202888 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 07 10:20:34.549979 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.549931 2577 apiserver.go:52] "Watching apiserver" Sep 07 10:20:34.562231 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.562188 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 07 10:20:34.564710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.564666 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal","openshift-multus/multus-additional-cni-plugins-94q5n","openshift-multus/multus-pbklr","openshift-multus/network-metrics-daemon-z5f5r","openshift-network-operator/iptables-alerter-6qrfb","kube-system/global-pull-secret-syncer-b5fbr","kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm","openshift-dns/node-resolver-lkh99","openshift-image-registry/node-ca-xz99t","openshift-network-diagnostics/network-check-target-97grq","openshift-ovn-kubernetes/ovnkube-node-4qpbw","kube-system/konnectivity-agent-hvgr4","openshift-cluster-node-tuning-operator/tuned-542h6"] Sep 07 10:20:34.566934 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.566893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.569335 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.568217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.569335 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.569166 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 07 10:20:34.569335 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.569167 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.569519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.569348 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhk84\"" Sep 07 10:20:34.569519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.569498 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.569610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.569552 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.571573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.570344 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 07 10:20:34.571573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.570443 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 07 10:20:34.571573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.570817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4zj8j\"" Sep 07 10:20:34.571813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.571718 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 07 10:20:34.572993 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.572969 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 07 10:20:34.573113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.572997 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fk67j\"" Sep 07 10:20:34.573176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.573156 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.573240 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.573209 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.573501 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.573485 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 07 10:20:34.573842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.573703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.573842 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.573785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:34.574031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.573855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.576166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.576024 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bj6mq\"" Sep 07 10:20:34.576166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.576042 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.576352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.576252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 07 10:20:34.576352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.576273 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.576707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.576569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.576707 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.576641 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:34.578734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.577747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.579118 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.579017 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.579883 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.579685 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.580760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.580477 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v7cms\"" Sep 07 10:20:34.580760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.580506 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.580916 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.580805 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.581487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.581268 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.581487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.581304 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sb4v5\"" Sep 07 10:20:34.581637 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.581502 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.583324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.582534 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.583324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.582954 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 07 10:20:34.583324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.583124 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 07 10:20:34.583324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.583256 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d9b8l\"" Sep 07 10:20:34.583582 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.583518 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 07 10:20:34.584063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.583938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:34.584063 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.584023 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:34.585175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.584388 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.585175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.584624 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2dt68\"" Sep 07 10:20:34.585175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.585061 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.585175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.585070 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 07 10:20:34.585572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.585553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.586913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.586946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-conf-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.586973 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-multus-certs\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m28t7\" (UniqueName: \"kubernetes.io/projected/ab990556-b0b6-42e1-ba02-6efd5c3b2541-kube-api-access-m28t7\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-socket-dir-parent\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwdr\" (UniqueName: \"kubernetes.io/projected/dfde7950-68a7-43e5-b58e-fb40830b9c22-kube-api-access-fqwdr\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-system-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-cnibin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-multus\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-etc-kubernetes\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfde7950-68a7-43e5-b58e-fb40830b9c22-host\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-cni-binary-copy\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-k8s-cni-cncf-io\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.587301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-kubelet\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-daemon-config\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8mvh\" (UniqueName: \"kubernetes.io/projected/4b2886c9-c467-41ea-aa26-db0748ea4400-kube-api-access-x8mvh\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab990556-b0b6-42e1-ba02-6efd5c3b2541-iptables-alerter-script\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab990556-b0b6-42e1-ba02-6efd5c3b2541-host-slash\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfde7950-68a7-43e5-b58e-fb40830b9c22-serviceca\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-bin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-hostroot\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-konnectivity-ca\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587549 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-agent-certs\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-netns\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9phn\" (UniqueName: \"kubernetes.io/projected/a9228703-6990-4c3f-b627-7597c3a0ff9e-kube-api-access-k9phn\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587686 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4gjq4\"" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.587911 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 07 10:20:34.588032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.588022 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 07 10:20:34.588886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.588275 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 07 10:20:34.588886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.588520 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 07 10:20:34.589108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.588943 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 07 10:20:34.589172 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.589110 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 07 10:20:34.606212 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.606178 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wphgl" Sep 07 10:20:34.616225 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.616191 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wphgl" Sep 07 10:20:34.641040 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.641013 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:34.673969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.673936 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 07 10:20:34.687914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.687882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-agent-certs\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.687929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysconfig\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.687961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.687986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12080a76-97b4-4c33-8a35-e44ec6689db9-ovn-node-metrics-cert\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9phn\" (UniqueName: \"kubernetes.io/projected/a9228703-6990-4c3f-b627-7597c3a0ff9e-kube-api-access-k9phn\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-kubelet\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688087 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-var-lib-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqh2x\" (UniqueName: \"kubernetes.io/projected/12080a76-97b4-4c33-8a35-e44ec6689db9-kube-api-access-nqh2x\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688138 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwdr\" (UniqueName: \"kubernetes.io/projected/dfde7950-68a7-43e5-b58e-fb40830b9c22-kube-api-access-fqwdr\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-system-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-cnibin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-modprobe-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-sys\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-socket-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-ovn\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688347 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-multus\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-system-cni-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-binary-copy\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-etc-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-netd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae088060-c534-4d00-a2c8-40beda7a51b8-tmp-dir\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-systemd\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-systemd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.688544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-lib-modules\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfde7950-68a7-43e5-b58e-fb40830b9c22-host\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-k8s-cni-cncf-io\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-daemon-config\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688586 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-dbus\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688618 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-device-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96sll\" (UniqueName: \"kubernetes.io/projected/90aa9f64-9471-4575-b857-ff227978ce51-kube-api-access-96sll\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8mvh\" (UniqueName: \"kubernetes.io/projected/4b2886c9-c467-41ea-aa26-db0748ea4400-kube-api-access-x8mvh\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab990556-b0b6-42e1-ba02-6efd5c3b2541-iptables-alerter-script\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab990556-b0b6-42e1-ba02-6efd5c3b2541-host-slash\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-hostroot\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688802 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-kubelet-config\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.688964 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-hostroot\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab990556-b0b6-42e1-ba02-6efd5c3b2541-host-slash\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-system-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-cnibin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.689408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-multus\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dfde7950-68a7-43e5-b58e-fb40830b9c22-host\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-registration-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-k8s-cni-cncf-io\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-slash\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-konnectivity-ca\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-kubernetes\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-sys-fs\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-config\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-netns\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689806 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-script-lib\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689811 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-netns\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.690310 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.689888 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.689920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.690001 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:35.189971106 +0000 UTC m=+3.047660401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-konnectivity-ca\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690015 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-daemon-config\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-conf-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-multus-certs\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-conf-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m28t7\" (UniqueName: \"kubernetes.io/projected/ab990556-b0b6-42e1-ba02-6efd5c3b2541-kube-api-access-m28t7\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab990556-b0b6-42e1-ba02-6efd5c3b2541-iptables-alerter-script\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-node-log\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-run-multus-certs\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-cni-dir\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-socket-dir-parent\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltr6\" (UniqueName: \"kubernetes.io/projected/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kube-api-access-6ltr6\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.691176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-multus-socket-dir-parent\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-systemd-units\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-etc-kubernetes\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-conf\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690486 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-var-lib-kubelet\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-etc-kubernetes\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-tmp\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-bin\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-env-overrides\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690568 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae088060-c534-4d00-a2c8-40beda7a51b8-hosts-file\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-cnibin\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-log-socket\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690695 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-host\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-cni-binary-copy\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-kubelet\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-kubelet\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqk9\" (UniqueName: \"kubernetes.io/projected/ae088060-c534-4d00-a2c8-40beda7a51b8-kube-api-access-2xqk9\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690920 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-run\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-etc-tuned\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.690996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfde7950-68a7-43e5-b58e-fb40830b9c22-serviceca\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-bin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnhb\" (UniqueName: \"kubernetes.io/projected/c3f94753-ba3e-411b-838c-0760af56c256-kube-api-access-jlnhb\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-netns\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9228703-6990-4c3f-b627-7597c3a0ff9e-host-var-lib-cni-bin\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691171 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a9228703-6990-4c3f-b627-7597c3a0ff9e-cni-binary-copy\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.692949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.691438 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dfde7950-68a7-43e5-b58e-fb40830b9c22-serviceca\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.698615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.698521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2fd4bdbf-9b94-4d89-899b-1b4fd6018932-agent-certs\") pod \"konnectivity-agent-hvgr4\" (UID: \"2fd4bdbf-9b94-4d89-899b-1b4fd6018932\") " pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.699375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.699302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8mvh\" (UniqueName: \"kubernetes.io/projected/4b2886c9-c467-41ea-aa26-db0748ea4400-kube-api-access-x8mvh\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:34.701114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.701088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m28t7\" (UniqueName: \"kubernetes.io/projected/ab990556-b0b6-42e1-ba02-6efd5c3b2541-kube-api-access-m28t7\") pod \"iptables-alerter-6qrfb\" (UID: \"ab990556-b0b6-42e1-ba02-6efd5c3b2541\") " pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.702035 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.702007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9phn\" (UniqueName: \"kubernetes.io/projected/a9228703-6990-4c3f-b627-7597c3a0ff9e-kube-api-access-k9phn\") pod \"multus-pbklr\" (UID: \"a9228703-6990-4c3f-b627-7597c3a0ff9e\") " pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.704665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.704193 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwdr\" (UniqueName: \"kubernetes.io/projected/dfde7950-68a7-43e5-b58e-fb40830b9c22-kube-api-access-fqwdr\") pod \"node-ca-xz99t\" (UID: \"dfde7950-68a7-43e5-b58e-fb40830b9c22\") " pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.723521 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.723455 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" event={"ID":"17f670fc079e23e62690da1f2f92b580","Type":"ContainerStarted","Data":"1ce9fbd0d219afa0cf3181304268376d65191960aea99a6dae73af69398b1c6e"} Sep 07 10:20:34.724953 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.724918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" event={"ID":"2d6c692e6d81acebe6a01d57406b92c1","Type":"ContainerStarted","Data":"c08530e0d5a2bdf56f1e97f3fdab51f409b6b8cdfbe2d0ec0257ce7341054f98"} Sep 07 10:20:34.791400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-modprobe-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.791400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-sys\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-socket-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-ovn\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-system-cni-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-binary-copy\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-etc-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-netd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae088060-c534-4d00-a2c8-40beda7a51b8-tmp-dir\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791538 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-systemd\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-systemd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-lib-modules\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-dbus\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791649 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-device-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.791691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96sll\" (UniqueName: \"kubernetes.io/projected/90aa9f64-9471-4575-b857-ff227978ce51-kube-api-access-96sll\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-kubelet-config\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-registration-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-slash\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-kubernetes\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791854 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-modprobe-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-sys-fs\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791914 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-sys\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791979 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-systemd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-system-cni-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-socket-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-systemd\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-config\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-run-ovn\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-d\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-lib-modules\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-dbus\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.792428 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae088060-c534-4d00-a2c8-40beda7a51b8-tmp-dir\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.792501 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:35.292482505 +0000 UTC m=+3.150171818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-script-lib\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-node-log\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792585 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-binary-copy\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792637 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-node-log\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-etc-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.792913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-device-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-config\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.792584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793043 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltr6\" (UniqueName: \"kubernetes.io/projected/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kube-api-access-6ltr6\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-systemd-units\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-conf\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-var-lib-kubelet\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-tmp\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb112766-fde8-43ff-9a61-36845bf0f30f-kubelet-config\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-bin\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-bin\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793215 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-env-overrides\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae088060-c534-4d00-a2c8-40beda7a51b8-hosts-file\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-ovnkube-script-lib\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-cnibin\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.793307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-log-socket\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-host\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqk9\" (UniqueName: \"kubernetes.io/projected/ae088060-c534-4d00-a2c8-40beda7a51b8-kube-api-access-2xqk9\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-run\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-etc-tuned\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793449 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnhb\" (UniqueName: \"kubernetes.io/projected/c3f94753-ba3e-411b-838c-0760af56c256-kube-api-access-jlnhb\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-netns\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793500 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysconfig\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12080a76-97b4-4c33-8a35-e44ec6689db9-ovn-node-metrics-cert\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793592 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-registration-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.791734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-cni-netd\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysctl-conf\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.793938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-slash\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-var-lib-kubelet\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-kubernetes\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-systemd-units\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.793594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-tuning-conf-dir\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-kubelet\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-var-lib-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqh2x\" (UniqueName: \"kubernetes.io/projected/12080a76-97b4-4c33-8a35-e44ec6689db9-kube-api-access-nqh2x\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-kubelet\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794221 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-host\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae088060-c534-4d00-a2c8-40beda7a51b8-hosts-file\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f94753-ba3e-411b-838c-0760af56c256-cnibin\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-var-lib-openvswitch\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-log-socket\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794450 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12080a76-97b4-4c33-8a35-e44ec6689db9-host-run-netns\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12080a76-97b4-4c33-8a35-e44ec6689db9-env-overrides\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.794630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.795358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-etc-sysconfig\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.795358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-sys-fs\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.795358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.794764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90aa9f64-9471-4575-b857-ff227978ce51-run\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.795358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.795240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3f94753-ba3e-411b-838c-0760af56c256-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.797980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.797938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-tmp\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.800391 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.799717 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:34.800391 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.799759 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:34.800391 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.799777 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:34.800391 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:34.799865 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:35.299844929 +0000 UTC m=+3.157534242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:34.800391 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.800280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90aa9f64-9471-4575-b857-ff227978ce51-etc-tuned\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.802310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.802027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96sll\" (UniqueName: \"kubernetes.io/projected/90aa9f64-9471-4575-b857-ff227978ce51-kube-api-access-96sll\") pod \"tuned-542h6\" (UID: \"90aa9f64-9471-4575-b857-ff227978ce51\") " pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.804865 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.804838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltr6\" (UniqueName: \"kubernetes.io/projected/617ee2d9-0957-4cb9-b3ee-f60bb9df8402-kube-api-access-6ltr6\") pod \"aws-ebs-csi-driver-node-q9gcm\" (UID: \"617ee2d9-0957-4cb9-b3ee-f60bb9df8402\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.807037 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.807006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnhb\" (UniqueName: \"kubernetes.io/projected/c3f94753-ba3e-411b-838c-0760af56c256-kube-api-access-jlnhb\") pod \"multus-additional-cni-plugins-94q5n\" (UID: \"c3f94753-ba3e-411b-838c-0760af56c256\") " pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.807198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.807170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqk9\" (UniqueName: \"kubernetes.io/projected/ae088060-c534-4d00-a2c8-40beda7a51b8-kube-api-access-2xqk9\") pod \"node-resolver-lkh99\" (UID: \"ae088060-c534-4d00-a2c8-40beda7a51b8\") " pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.809586 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.809483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqh2x\" (UniqueName: \"kubernetes.io/projected/12080a76-97b4-4c33-8a35-e44ec6689db9-kube-api-access-nqh2x\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.809724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.809586 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12080a76-97b4-4c33-8a35-e44ec6689db9-ovn-node-metrics-cert\") pod \"ovnkube-node-4qpbw\" (UID: \"12080a76-97b4-4c33-8a35-e44ec6689db9\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.882980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.882939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xz99t" Sep 07 10:20:34.893677 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.893643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:34.903085 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.903046 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pbklr" Sep 07 10:20:34.920070 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.920033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6qrfb" Sep 07 10:20:34.925454 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.925212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lkh99" Sep 07 10:20:34.935500 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.935462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-542h6" Sep 07 10:20:34.945016 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.944978 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-94q5n" Sep 07 10:20:34.958150 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.958101 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" Sep 07 10:20:34.965629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.965575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:34.970542 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:34.970514 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 07 10:20:35.198078 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.197957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:35.198281 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.198154 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:35.198281 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.198232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:36.198213213 +0000 UTC m=+4.055902511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:35.298450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.298401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:35.298692 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.298588 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:35.298692 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.298673 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:36.298655954 +0000 UTC m=+4.156345264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:35.399861 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.399758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:35.400245 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.400098 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:35.400245 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.400153 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:35.400245 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.400175 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:35.400427 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.400309 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:36.400267938 +0000 UTC m=+4.257957238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:35.618643 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.618572 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:15:35 +0000 UTC" deadline="2028-02-07 13:12:04.495694449 +0000 UTC" Sep 07 10:20:35.618643 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.618640 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12434h51m28.877059422s" Sep 07 10:20:35.717101 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:35.717060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:35.717297 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:35.717216 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:35.950215 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.950151 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd4bdbf_9b94_4d89_899b_1b4fd6018932.slice/crio-4f766314221ca849188d1149d27d450fe904345418881e66fd22b05459af384b WatchSource:0}: Error finding container 4f766314221ca849188d1149d27d450fe904345418881e66fd22b05459af384b: Status 404 returned error can't find the container with id 4f766314221ca849188d1149d27d450fe904345418881e66fd22b05459af384b Sep 07 10:20:35.954491 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.954371 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab990556_b0b6_42e1_ba02_6efd5c3b2541.slice/crio-4767d9375c434d139a746d7c170461f561d8a717708f56a812f7dcd371617d86 WatchSource:0}: Error finding container 4767d9375c434d139a746d7c170461f561d8a717708f56a812f7dcd371617d86: Status 404 returned error can't find the container with id 4767d9375c434d139a746d7c170461f561d8a717708f56a812f7dcd371617d86 Sep 07 10:20:35.957220 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.957188 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfde7950_68a7_43e5_b58e_fb40830b9c22.slice/crio-277033719c59b095cf49bc59cd7ce765eefb10bdff1734994f5021edba7262a8 WatchSource:0}: Error finding container 277033719c59b095cf49bc59cd7ce765eefb10bdff1734994f5021edba7262a8: Status 404 returned error can't find the container with id 277033719c59b095cf49bc59cd7ce765eefb10bdff1734994f5021edba7262a8 Sep 07 10:20:35.959955 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.959925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f94753_ba3e_411b_838c_0760af56c256.slice/crio-4234368d9e6f6070c71ccb2bf86a6e3056eaf0ad2d444362491f778eed09d42e WatchSource:0}: Error finding container 4234368d9e6f6070c71ccb2bf86a6e3056eaf0ad2d444362491f778eed09d42e: Status 404 returned error can't find the container with id 4234368d9e6f6070c71ccb2bf86a6e3056eaf0ad2d444362491f778eed09d42e Sep 07 10:20:35.961261 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.961236 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aa9f64_9471_4575_b857_ff227978ce51.slice/crio-c2f8488391c9bee69fbea650ee76ebff3338a22e176865b42949c1bacfb77512 WatchSource:0}: Error finding container c2f8488391c9bee69fbea650ee76ebff3338a22e176865b42949c1bacfb77512: Status 404 returned error can't find the container with id c2f8488391c9bee69fbea650ee76ebff3338a22e176865b42949c1bacfb77512 Sep 07 10:20:35.963827 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.963562 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae088060_c534_4d00_a2c8_40beda7a51b8.slice/crio-28b36ff9e982bf83e4b383242b153483837e22a20da30bc7faf36dc783269d8e WatchSource:0}: Error finding container 28b36ff9e982bf83e4b383242b153483837e22a20da30bc7faf36dc783269d8e: Status 404 returned error can't find the container with id 28b36ff9e982bf83e4b383242b153483837e22a20da30bc7faf36dc783269d8e Sep 07 10:20:35.966916 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:20:35.966370 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12080a76_97b4_4c33_8a35_e44ec6689db9.slice/crio-0cceff615f4e3b2d40ad4a5ebef3d901f94d6901ad3abb51652271811b30f04e WatchSource:0}: Error finding container 0cceff615f4e3b2d40ad4a5ebef3d901f94d6901ad3abb51652271811b30f04e: Status 404 returned error can't find the container with id 0cceff615f4e3b2d40ad4a5ebef3d901f94d6901ad3abb51652271811b30f04e Sep 07 10:20:36.208336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.208296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:36.208567 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.208437 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:36.208567 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.208514 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:38.208493269 +0000 UTC m=+6.066182571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:36.308914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.308859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:36.309135 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.309044 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:36.309135 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.309129 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:38.309110031 +0000 UTC m=+6.166799350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:36.409541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.409488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:36.409773 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.409749 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:36.409773 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.409774 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:36.409927 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.409904 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:36.410047 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.410017 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:38.40999303 +0000 UTC m=+6.267682339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:36.619854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.619775 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:15:35 +0000 UTC" deadline="2028-03-12 14:20:24.697239773 +0000 UTC" Sep 07 10:20:36.619854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.619850 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13251h59m48.077394551s" Sep 07 10:20:36.719812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.719773 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:36.720038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.719832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:36.720038 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.719930 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:36.720412 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:36.720381 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:36.734778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.734739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" event={"ID":"617ee2d9-0957-4cb9-b3ee-f60bb9df8402","Type":"ContainerStarted","Data":"75f4cc86f5f1aa7823a2ea65d32e454fdf30ca8691113cdd9354c1d977c4f968"} Sep 07 10:20:36.742092 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.742048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lkh99" event={"ID":"ae088060-c534-4d00-a2c8-40beda7a51b8","Type":"ContainerStarted","Data":"28b36ff9e982bf83e4b383242b153483837e22a20da30bc7faf36dc783269d8e"} Sep 07 10:20:36.745827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.745786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pbklr" event={"ID":"a9228703-6990-4c3f-b627-7597c3a0ff9e","Type":"ContainerStarted","Data":"35f0f6b30eabbc4ba8550775a435c082ecd8ccdf0c72f8b8851398c8496ca11b"} Sep 07 10:20:36.750006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.749969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-542h6" event={"ID":"90aa9f64-9471-4575-b857-ff227978ce51","Type":"ContainerStarted","Data":"c2f8488391c9bee69fbea650ee76ebff3338a22e176865b42949c1bacfb77512"} Sep 07 10:20:36.753285 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.753224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz99t" event={"ID":"dfde7950-68a7-43e5-b58e-fb40830b9c22","Type":"ContainerStarted","Data":"277033719c59b095cf49bc59cd7ce765eefb10bdff1734994f5021edba7262a8"} Sep 07 10:20:36.756808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.756771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qrfb" event={"ID":"ab990556-b0b6-42e1-ba02-6efd5c3b2541","Type":"ContainerStarted","Data":"4767d9375c434d139a746d7c170461f561d8a717708f56a812f7dcd371617d86"} Sep 07 10:20:36.767252 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.767098 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvgr4" event={"ID":"2fd4bdbf-9b94-4d89-899b-1b4fd6018932","Type":"ContainerStarted","Data":"4f766314221ca849188d1149d27d450fe904345418881e66fd22b05459af384b"} Sep 07 10:20:36.778098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.778052 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" event={"ID":"17f670fc079e23e62690da1f2f92b580","Type":"ContainerStarted","Data":"9e62e0c647cf3946b055597542bd49d9e16662ea9ffb9ad2f49f95aab2a71645"} Sep 07 10:20:36.781904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.781865 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d6c692e6d81acebe6a01d57406b92c1" containerID="94be22d72424772631af03040b5d7634f8e136a458cec13ae85f48568d8e751a" exitCode=0 Sep 07 10:20:36.782397 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.782365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" event={"ID":"2d6c692e6d81acebe6a01d57406b92c1","Type":"ContainerDied","Data":"94be22d72424772631af03040b5d7634f8e136a458cec13ae85f48568d8e751a"} Sep 07 10:20:36.805635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.805596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"0cceff615f4e3b2d40ad4a5ebef3d901f94d6901ad3abb51652271811b30f04e"} Sep 07 10:20:36.810337 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.810287 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-95.ec2.internal" podStartSLOduration=2.810269644 podStartE2EDuration="2.810269644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:36.793556867 +0000 UTC m=+4.651246188" watchObservedRunningTime="2026-09-07 10:20:36.810269644 +0000 UTC m=+4.667958963" Sep 07 10:20:36.823504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:36.823463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerStarted","Data":"4234368d9e6f6070c71ccb2bf86a6e3056eaf0ad2d444362491f778eed09d42e"} Sep 07 10:20:37.717466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:37.717403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:37.718003 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:37.717585 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:37.890850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:37.890761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" event={"ID":"2d6c692e6d81acebe6a01d57406b92c1","Type":"ContainerStarted","Data":"2582903e7e39b7a50a0ecec1251aaf71d73964c2112f2e082d95db194c29d281"} Sep 07 10:20:38.225564 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:38.225522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:38.225811 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.225715 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:38.225811 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.225793 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:42.225770125 +0000 UTC m=+10.083459447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:38.326842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:38.326796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:38.327058 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.327042 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:38.327132 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.327113 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:42.327093452 +0000 UTC m=+10.184782764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:38.427681 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:38.427634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:38.427893 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.427833 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:38.427893 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.427857 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:38.427893 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.427869 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:38.428063 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.427933 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:42.427912616 +0000 UTC m=+10.285601935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:38.719523 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:38.719476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:38.720094 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.719644 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:38.720094 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:38.719733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:38.720094 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:38.719813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:39.717343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:39.717307 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:39.717564 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:39.717437 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:40.719768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:40.719338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:40.719768 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:40.719445 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:40.719768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:40.719726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:40.720326 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:40.719845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:41.717147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:41.717065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:41.717330 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:41.717233 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:42.261927 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:42.261885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:42.262412 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.262079 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:42.262412 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.262167 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:50.26214436 +0000 UTC m=+18.119833677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:42.363422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:42.362832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:42.363422 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.363030 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:42.363422 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.363097 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:20:50.363078642 +0000 UTC m=+18.220767956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:42.463968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:42.463926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:42.464174 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.464136 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:42.464174 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.464159 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:42.464174 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.464172 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:42.464330 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.464234 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:20:50.464215317 +0000 UTC m=+18.321904625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:42.718228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:42.718201 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:42.718383 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.718311 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:42.718859 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:42.718693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:42.718859 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:42.718786 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:43.717732 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:43.717622 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:43.718629 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:43.717912 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:44.717300 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:44.717257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:44.717504 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:44.717404 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:44.717876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:44.717853 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:44.718278 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:44.717966 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:45.716751 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:45.716723 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:45.716937 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:45.716855 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:46.716883 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:46.716844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:46.717323 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:46.716898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:46.717323 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:46.717006 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:46.717323 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:46.717127 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:47.716909 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:47.716871 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:47.717413 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:47.716995 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:48.716762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:48.716727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:48.716941 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:48.716742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:48.716941 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:48.716859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:48.717355 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:48.716952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:49.717418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:49.717371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:49.717922 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:49.717519 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:50.328809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:50.328764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:50.329067 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.328879 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:50.329067 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.328934 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:06.328919319 +0000 UTC m=+34.186608614 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:20:50.429497 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:50.429460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:50.429680 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.429612 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:50.429723 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.429681 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:21:06.429662893 +0000 UTC m=+34.287352192 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:20:50.530236 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:50.530194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:50.530417 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.530322 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:20:50.530417 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.530338 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:20:50.530417 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.530348 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:50.530417 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.530402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:06.530383256 +0000 UTC m=+34.388072550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:20:50.717136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:50.717043 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:50.717317 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.717203 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:50.717317 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:50.717251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:50.717404 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:50.717376 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:51.716809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:51.716774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:51.717355 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:51.716917 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:52.717869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:52.717824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:52.718355 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:52.717949 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:52.718355 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:52.718008 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:52.718355 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:52.718123 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:53.717482 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:53.717432 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:53.717708 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:53.717602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:54.717147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.717027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:54.717147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.717074 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:54.717857 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:54.717165 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:54.717857 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:54.717627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:54.929937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.929745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" event={"ID":"617ee2d9-0957-4cb9-b3ee-f60bb9df8402","Type":"ContainerStarted","Data":"d53d6e0ab65786dd3b88091b1e553413decf8db19ce22804214f10141814ae9a"} Sep 07 10:20:54.931281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.931249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lkh99" event={"ID":"ae088060-c534-4d00-a2c8-40beda7a51b8","Type":"ContainerStarted","Data":"6d520e5909d515dc40b63db1849dea0e9f3908dc6bdd72aac43bc26639abff6e"} Sep 07 10:20:54.932880 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.932851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pbklr" event={"ID":"a9228703-6990-4c3f-b627-7597c3a0ff9e","Type":"ContainerStarted","Data":"e85209b79fb45b0b548d030b79cf51aac93b238bdb07ea33446cd94e7ee9e1f7"} Sep 07 10:20:54.936775 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.934695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-542h6" event={"ID":"90aa9f64-9471-4575-b857-ff227978ce51","Type":"ContainerStarted","Data":"3e51403d70fdcec0b06e8a45b7ba23d80d191b24c77aa785b009f0457e0d40c7"} Sep 07 10:20:54.938328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.938307 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xz99t" event={"ID":"dfde7950-68a7-43e5-b58e-fb40830b9c22","Type":"ContainerStarted","Data":"80040cefec9e762c1666fadb1cdb350beb3bf23fc30c7de4e618070756ccedec"} Sep 07 10:20:54.939623 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.939601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvgr4" event={"ID":"2fd4bdbf-9b94-4d89-899b-1b4fd6018932","Type":"ContainerStarted","Data":"a160f02c50ac931ed7cb94557676616f964285e00e18b5804fa3aff31e0aa542"} Sep 07 10:20:54.941829 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.941809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"62ab6e68601653af029ad916fa1b629bf4f3884afda1441e58479a8a5e3fc188"} Sep 07 10:20:54.941923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.941875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"81a85ce0ce0087e8f634cedef616b8aa395eb12accffdcbd81bffa140e2ef9e6"} Sep 07 10:20:54.941923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.941906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"d7904fe001cbced76d6c703caafd8ac5d7347d4ef7cf2527ea588ed9bb3fb418"} Sep 07 10:20:54.941998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.941948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"a75b8ae14bac206c6d3560b09728e85a44ce4fa96893f1b24d58315dd0add177"} Sep 07 10:20:54.943239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.943216 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="29761abb86386d8ff4f45d80f1eb13695d59cc64a9cb57a5af808e78ac3161c9" exitCode=0 Sep 07 10:20:54.943313 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.943249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"29761abb86386d8ff4f45d80f1eb13695d59cc64a9cb57a5af808e78ac3161c9"} Sep 07 10:20:54.944347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.944313 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-95.ec2.internal" podStartSLOduration=20.944301539 podStartE2EDuration="20.944301539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:37.906088168 +0000 UTC m=+5.763777490" watchObservedRunningTime="2026-09-07 10:20:54.944301539 +0000 UTC m=+22.801990854" Sep 07 10:20:54.944715 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.944691 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lkh99" podStartSLOduration=3.4935284859999998 podStartE2EDuration="21.944684803s" totalImagesPullingTime="18.451156317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:54.944248839 +0000 UTC m=+22.801938154" watchObservedRunningTime="2026-09-07 10:20:54.944684803 +0000 UTC m=+22.802374120" Sep 07 10:20:54.959209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.959161 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pbklr" podStartSLOduration=3.445713522 podStartE2EDuration="21.959141404s" totalImagesPullingTime="18.513427882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:54.958872863 +0000 UTC m=+22.816562178" watchObservedRunningTime="2026-09-07 10:20:54.959141404 +0000 UTC m=+22.816830723" Sep 07 10:20:54.973280 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.973237 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-542h6" podStartSLOduration=3.499498571 podStartE2EDuration="21.973218989s" totalImagesPullingTime="18.473720418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:54.972862311 +0000 UTC m=+22.830551627" watchObservedRunningTime="2026-09-07 10:20:54.973218989 +0000 UTC m=+22.830908297" Sep 07 10:20:54.984338 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.984296 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xz99t" podStartSLOduration=3.52635709 podStartE2EDuration="21.984281494s" totalImagesPullingTime="18.457924404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:54.984185358 +0000 UTC m=+22.841874674" watchObservedRunningTime="2026-09-07 10:20:54.984281494 +0000 UTC m=+22.841970810" Sep 07 10:20:54.996174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:54.996112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hvgr4" podStartSLOduration=7.575752093 podStartE2EDuration="21.996091321s" totalImagesPullingTime="14.420339228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:54.995355602 +0000 UTC m=+22.853044918" watchObservedRunningTime="2026-09-07 10:20:54.996091321 +0000 UTC m=+22.853780662" Sep 07 10:20:55.647886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.647848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:55.716772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.716736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:55.716981 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:55.716871 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:55.948132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.948103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" event={"ID":"617ee2d9-0957-4cb9-b3ee-f60bb9df8402","Type":"ContainerStarted","Data":"b568b5a3df3eb539677f0d7569e3a3461d58998d6297b934896947c18f8f4809"} Sep 07 10:20:55.949760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.949731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6qrfb" event={"ID":"ab990556-b0b6-42e1-ba02-6efd5c3b2541","Type":"ContainerStarted","Data":"283de720898941f77230784902cd7c3dd7eda1a5f0a9a938e6ddece992a571b3"} Sep 07 10:20:55.954216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.954186 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 07 10:20:55.955683 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.954851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"57fd59a76fb6910a8532015f77dd4d39b192f8c7f50b0fb74d3e6df9679737de"} Sep 07 10:20:55.955683 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.954943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"bdd7521cbe3503c6ee0ff445d11dbfc7f7db7b6e945bfe74455afeeae6f0951c"} Sep 07 10:20:55.963338 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:55.963292 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6qrfb" podStartSLOduration=4.502491667 podStartE2EDuration="22.96327507s" totalImagesPullingTime="18.460783403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:55.96290134 +0000 UTC m=+23.820590655" watchObservedRunningTime="2026-09-07 10:20:55.96327507 +0000 UTC m=+23.820964385" Sep 07 10:20:56.658079 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:56.657955 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T10:20:55.954206574Z","UUID":"45c0e438-9c88-413b-a216-21488161fbe9","Handler":null,"Name":"","Endpoint":""} Sep 07 10:20:56.660118 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:56.660087 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 07 10:20:56.660259 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:56.660129 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 07 10:20:56.716753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:56.716723 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:56.716932 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:56.716846 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:56.717192 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:56.716724 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:56.717330 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:56.717277 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:57.716849 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:57.716808 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:57.717339 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:57.716952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:57.961670 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:57.961627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" event={"ID":"617ee2d9-0957-4cb9-b3ee-f60bb9df8402","Type":"ContainerStarted","Data":"a09c6f66fd3a27b119f7fe0ea95ce355f15458e95c37d7466e2c020a2605b95d"} Sep 07 10:20:57.965110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:57.965087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"cd3fbdf0576afee1b44be850461c8e8e11effa5b2abda06069d32875dd865ef9"} Sep 07 10:20:57.979665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:57.979621 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9gcm" podStartSLOduration=4.24003162 podStartE2EDuration="24.979606813s" totalImagesPullingTime="20.739575193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:57.97932437 +0000 UTC m=+25.837013686" watchObservedRunningTime="2026-09-07 10:20:57.979606813 +0000 UTC m=+25.837296125" Sep 07 10:20:58.716716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:58.716675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:20:58.716716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:58.716706 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:20:58.717266 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:58.716819 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:20:58.717266 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:58.717215 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:20:58.754724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:58.754691 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:58.755440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:58.755411 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:58.968619 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:58.968577 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hvgr4" Sep 07 10:20:59.717533 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.717355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:20:59.718029 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:20:59.717677 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:20:59.973448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.973418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" event={"ID":"12080a76-97b4-4c33-8a35-e44ec6689db9","Type":"ContainerStarted","Data":"c52a74f1b0ecd313cc1fe9fa76f6b2185b74a950745e482d23913e50cfb10d61"} Sep 07 10:20:59.973837 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.973804 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:59.973944 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.973885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:59.973944 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.973938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:59.975576 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.975551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerStarted","Data":"4192f18e1680cdeeabf6abf218c2a554c5bac643d8283cd440d2ea863afe96b7"} Sep 07 10:20:59.994255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.993051 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:59.998304 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.998276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:20:59.998425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:20:59.998296 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" podStartSLOduration=8.493577895 podStartE2EDuration="26.998281332s" totalImagesPullingTime="18.504703437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:59.997825865 +0000 UTC m=+27.855515194" watchObservedRunningTime="2026-09-07 10:20:59.998281332 +0000 UTC m=+27.855970647" Sep 07 10:21:00.717494 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:00.717446 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:00.717494 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:00.717493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:00.718290 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:00.717574 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:00.718290 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:00.717709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:00.979443 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:00.979406 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="4192f18e1680cdeeabf6abf218c2a554c5bac643d8283cd440d2ea863afe96b7" exitCode=0 Sep 07 10:21:00.979669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:00.979494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"4192f18e1680cdeeabf6abf218c2a554c5bac643d8283cd440d2ea863afe96b7"} Sep 07 10:21:01.717130 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:01.716953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:01.717252 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:01.717224 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:21:01.984438 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:01.984404 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="cfff4beed4724870fb5c052e1763a75acef463cf43cac11af3c48293dd8fcb51" exitCode=0 Sep 07 10:21:01.984932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:01.984497 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"cfff4beed4724870fb5c052e1763a75acef463cf43cac11af3c48293dd8fcb51"} Sep 07 10:21:02.717412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:02.717385 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:02.717556 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:02.717469 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:02.717629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:02.717576 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:02.717728 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:02.717705 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:02.988825 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:02.988786 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="79e173a3173eb89f90d11bc1820fb8b77367ee78376ffe73ce576d34e2ebdce1" exitCode=0 Sep 07 10:21:02.989196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:02.988853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"79e173a3173eb89f90d11bc1820fb8b77367ee78376ffe73ce576d34e2ebdce1"} Sep 07 10:21:03.716671 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:03.716638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:03.716861 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:03.716770 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:21:04.717568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:04.717533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:04.717960 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:04.717539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:04.717960 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:04.717673 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:04.717960 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:04.717748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:05.566496 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.566453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b5fbr"] Sep 07 10:21:05.566701 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.566613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:05.566758 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:05.566735 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:05.574382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.574340 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-97grq"] Sep 07 10:21:05.574628 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.574464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:05.574628 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:05.574571 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:05.575683 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.575601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5f5r"] Sep 07 10:21:05.575789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:05.575707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:05.575850 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:05.575827 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:21:06.359721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:06.359269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:06.359721 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.359424 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:21:06.359721 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.359485 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs podName:4b2886c9-c467-41ea-aa26-db0748ea4400 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:38.359469655 +0000 UTC m=+66.217158962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs") pod "network-metrics-daemon-z5f5r" (UID: "4b2886c9-c467-41ea-aa26-db0748ea4400") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 07 10:21:06.460569 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:06.460529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:06.460745 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.460696 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 07 10:21:06.460790 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.460764 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret podName:eb112766-fde8-43ff-9a61-36845bf0f30f nodeName:}" failed. No retries permitted until 2026-09-07 10:21:38.460748139 +0000 UTC m=+66.318437433 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret") pod "global-pull-secret-syncer-b5fbr" (UID: "eb112766-fde8-43ff-9a61-36845bf0f30f") : object "kube-system"/"original-pull-secret" not registered Sep 07 10:21:06.561842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:06.561809 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:06.561996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.561953 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 07 10:21:06.561996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.561971 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 07 10:21:06.561996 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.561980 2577 projected.go:196] Error preparing data for projected volume kube-api-access-8jlc6 for pod openshift-network-diagnostics/network-check-target-97grq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:21:06.562092 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:06.562030 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6 podName:566e9ed2-77d3-4caf-82e5-5ceae47df598 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:38.5620171 +0000 UTC m=+66.419706397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8jlc6" (UniqueName: "kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6") pod "network-check-target-97grq" (UID: "566e9ed2-77d3-4caf-82e5-5ceae47df598") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 07 10:21:07.717244 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:07.717200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:07.717244 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:07.717224 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:07.717803 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:07.717221 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:07.717803 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:07.717360 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:21:07.717803 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:07.717421 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:07.717803 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:07.717518 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:09.008209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:09.007955 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="3e514eef28af7e8f968e7a9b99ee0acc66fb305b538389cbdc8181ce238b723e" exitCode=0 Sep 07 10:21:09.008734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:09.008021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"3e514eef28af7e8f968e7a9b99ee0acc66fb305b538389cbdc8181ce238b723e"} Sep 07 10:21:09.717486 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:09.717398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:09.717661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:09.717415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:09.717661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:09.717526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:09.717661 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:09.717526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b5fbr" podUID="eb112766-fde8-43ff-9a61-36845bf0f30f" Sep 07 10:21:09.717661 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:09.717637 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-z5f5r" podUID="4b2886c9-c467-41ea-aa26-db0748ea4400" Sep 07 10:21:09.717800 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:09.717696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-97grq" podUID="566e9ed2-77d3-4caf-82e5-5ceae47df598" Sep 07 10:21:10.013658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.013625 2577 generic.go:356] "Generic (PLEG): container finished" podID="c3f94753-ba3e-411b-838c-0760af56c256" containerID="a3722bb7cade96ef5dee2b6924af5e652d89167eb59b96a74ec1e7e61eefcf96" exitCode=0 Sep 07 10:21:10.014036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.013695 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerDied","Data":"a3722bb7cade96ef5dee2b6924af5e652d89167eb59b96a74ec1e7e61eefcf96"} Sep 07 10:21:10.456296 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.456263 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-95.ec2.internal" event="NodeReady" Sep 07 10:21:10.456494 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.456426 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 07 10:21:10.502647 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.502603 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xbrkj"] Sep 07 10:21:10.524310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.524279 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cdcwk"] Sep 07 10:21:10.524466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.524399 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.526432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.526397 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 07 10:21:10.526603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.526451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 07 10:21:10.526603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.526449 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xz5mc\"" Sep 07 10:21:10.526603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.526452 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 07 10:21:10.526603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.526568 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 07 10:21:10.536523 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.536489 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cdcwk"] Sep 07 10:21:10.536669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.536532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbrkj"] Sep 07 10:21:10.536669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.536612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.538699 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.538679 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 07 10:21:10.538817 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.538728 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-594j9\"" Sep 07 10:21:10.538817 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.538757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 07 10:21:10.538817 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.538813 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 07 10:21:10.601013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.600975 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fn7dn"] Sep 07 10:21:10.605792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.605763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.607770 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.607749 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 07 10:21:10.607951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.607935 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 07 10:21:10.608361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.608343 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vkvfh\"" Sep 07 10:21:10.611802 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.611781 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fn7dn"] Sep 07 10:21:10.692203 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692162 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.692203 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz24d\" (UniqueName: \"kubernetes.io/projected/af0133f0-b811-47d0-b008-bab7a80516b9-kube-api-access-lz24d\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c139569-281b-487b-9851-616f975116b1-metrics-tls\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0133f0-b811-47d0-b008-bab7a80516b9-cert\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c139569-281b-487b-9851-616f975116b1-config-volume\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqd5\" (UniqueName: \"kubernetes.io/projected/1c139569-281b-487b-9851-616f975116b1-kube-api-access-rhqd5\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-crio-socket\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692417 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.692463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-data-volume\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.692735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l2d\" (UniqueName: \"kubernetes.io/projected/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-api-access-52l2d\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.692735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.692522 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c139569-281b-487b-9851-616f975116b1-tmp-dir\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.793436 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.793436 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz24d\" (UniqueName: \"kubernetes.io/projected/af0133f0-b811-47d0-b008-bab7a80516b9-kube-api-access-lz24d\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.793721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c139569-281b-487b-9851-616f975116b1-metrics-tls\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.793721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0133f0-b811-47d0-b008-bab7a80516b9-cert\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.793721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c139569-281b-487b-9851-616f975116b1-config-volume\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.793876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqd5\" (UniqueName: \"kubernetes.io/projected/1c139569-281b-487b-9851-616f975116b1-kube-api-access-rhqd5\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.793876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-crio-socket\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.793876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.793876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-data-volume\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.793876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l2d\" (UniqueName: \"kubernetes.io/projected/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-api-access-52l2d\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.794089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.793890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c139569-281b-487b-9851-616f975116b1-tmp-dir\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.794089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.794012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-crio-socket\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.794262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.794237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1c139569-281b-487b-9851-616f975116b1-config-volume\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.794384 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.794240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c139569-281b-487b-9851-616f975116b1-tmp-dir\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.794384 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.794287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-data-volume\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.794936 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.794706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.797772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.797744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.797876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.797747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c139569-281b-487b-9851-616f975116b1-metrics-tls\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.797958 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.797933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af0133f0-b811-47d0-b008-bab7a80516b9-cert\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.801159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.801130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l2d\" (UniqueName: \"kubernetes.io/projected/0b9a1cae-7394-4c14-a23b-4c1d35f5d258-kube-api-access-52l2d\") pod \"insights-runtime-extractor-xbrkj\" (UID: \"0b9a1cae-7394-4c14-a23b-4c1d35f5d258\") " pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.801793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.801767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqd5\" (UniqueName: \"kubernetes.io/projected/1c139569-281b-487b-9851-616f975116b1-kube-api-access-rhqd5\") pod \"dns-default-fn7dn\" (UID: \"1c139569-281b-487b-9851-616f975116b1\") " pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:10.801951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.801931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz24d\" (UniqueName: \"kubernetes.io/projected/af0133f0-b811-47d0-b008-bab7a80516b9-kube-api-access-lz24d\") pod \"ingress-canary-cdcwk\" (UID: \"af0133f0-b811-47d0-b008-bab7a80516b9\") " pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.833870 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.833831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xbrkj" Sep 07 10:21:10.844862 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.844835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cdcwk" Sep 07 10:21:10.915876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:10.915464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:11.019937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.019746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-94q5n" event={"ID":"c3f94753-ba3e-411b-838c-0760af56c256","Type":"ContainerStarted","Data":"d59ce82028f62f672186938348239c6c67e527c227769a90c8e6224f330e0ee3"} Sep 07 10:21:11.040691 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.040634 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-94q5n" podStartSLOduration=7.9780325 podStartE2EDuration="38.040617362s" totalImagesPullingTime="30.062584862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:11.039303289 +0000 UTC m=+38.896992607" watchObservedRunningTime="2026-09-07 10:21:11.040617362 +0000 UTC m=+38.898306677" Sep 07 10:21:11.136359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.136259 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cdcwk"] Sep 07 10:21:11.139463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.139435 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xbrkj"] Sep 07 10:21:11.140200 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:11.140146 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0133f0_b811_47d0_b008_bab7a80516b9.slice/crio-0d86a9d85947a987821e278ddda8ace56c6b06fa8bcaa1d149329b81e7c57c12 WatchSource:0}: Error finding container 0d86a9d85947a987821e278ddda8ace56c6b06fa8bcaa1d149329b81e7c57c12: Status 404 returned error can't find the container with id 0d86a9d85947a987821e278ddda8ace56c6b06fa8bcaa1d149329b81e7c57c12 Sep 07 10:21:11.142858 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:11.142690 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9a1cae_7394_4c14_a23b_4c1d35f5d258.slice/crio-3fd503bdf605b7e055023ac7fcb00fbdd8c3350e6f4d8355d34cc412e66b0257 WatchSource:0}: Error finding container 3fd503bdf605b7e055023ac7fcb00fbdd8c3350e6f4d8355d34cc412e66b0257: Status 404 returned error can't find the container with id 3fd503bdf605b7e055023ac7fcb00fbdd8c3350e6f4d8355d34cc412e66b0257 Sep 07 10:21:11.203266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.203243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fn7dn"] Sep 07 10:21:11.206638 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:11.206611 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c139569_281b_487b_9851_616f975116b1.slice/crio-9b66c91c1adc732355b27ee39632bade8b83ae0f931f92559b05b700ed13b3d4 WatchSource:0}: Error finding container 9b66c91c1adc732355b27ee39632bade8b83ae0f931f92559b05b700ed13b3d4: Status 404 returned error can't find the container with id 9b66c91c1adc732355b27ee39632bade8b83ae0f931f92559b05b700ed13b3d4 Sep 07 10:21:11.717235 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.717196 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:11.717407 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.717238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:11.717407 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.717210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:11.719336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.719313 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 10:21:11.719484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.719411 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 10:21:11.720009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.719990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 10:21:11.720082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.720029 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 10:21:11.720082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.720037 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbx74\"" Sep 07 10:21:11.720082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:11.720048 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jrgm\"" Sep 07 10:21:12.024791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:12.024757 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbrkj" event={"ID":"0b9a1cae-7394-4c14-a23b-4c1d35f5d258","Type":"ContainerStarted","Data":"6368132661f45fa580c2557631e322fcbb82d51c909e0229f1c11f7d1a97e67e"} Sep 07 10:21:12.025361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:12.024832 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbrkj" event={"ID":"0b9a1cae-7394-4c14-a23b-4c1d35f5d258","Type":"ContainerStarted","Data":"3fd503bdf605b7e055023ac7fcb00fbdd8c3350e6f4d8355d34cc412e66b0257"} Sep 07 10:21:12.026904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:12.026875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn7dn" event={"ID":"1c139569-281b-487b-9851-616f975116b1","Type":"ContainerStarted","Data":"9b66c91c1adc732355b27ee39632bade8b83ae0f931f92559b05b700ed13b3d4"} Sep 07 10:21:12.028531 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:12.028502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cdcwk" event={"ID":"af0133f0-b811-47d0-b008-bab7a80516b9","Type":"ContainerStarted","Data":"0d86a9d85947a987821e278ddda8ace56c6b06fa8bcaa1d149329b81e7c57c12"} Sep 07 10:21:13.033646 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.033577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbrkj" event={"ID":"0b9a1cae-7394-4c14-a23b-4c1d35f5d258","Type":"ContainerStarted","Data":"db9d7c4e076cfd0ea8395c5642761f70accc6af32c909d95750e4b49603d7a8e"} Sep 07 10:21:13.094378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.094339 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d86sm"] Sep 07 10:21:13.120512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.120479 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d86sm"] Sep 07 10:21:13.120690 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.120624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.123266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123238 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b47v9\"" Sep 07 10:21:13.123410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123344 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 07 10:21:13.123410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123353 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 07 10:21:13.123410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123365 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 07 10:21:13.123410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123238 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 07 10:21:13.123579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.123432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 07 10:21:13.212753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.212714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2f4\" (UniqueName: \"kubernetes.io/projected/1e94ffbd-080c-4f67-8746-f3e8b192d286-kube-api-access-cr2f4\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.212940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.212766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e94ffbd-080c-4f67-8746-f3e8b192d286-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.212940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.212880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.212940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.212922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.313392 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.313302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.313392 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.313358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.313392 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.313388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2f4\" (UniqueName: \"kubernetes.io/projected/1e94ffbd-080c-4f67-8746-f3e8b192d286-kube-api-access-cr2f4\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.313640 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.313441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e94ffbd-080c-4f67-8746-f3e8b192d286-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.313640 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:13.313482 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 07 10:21:13.313640 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:13.313558 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls podName:1e94ffbd-080c-4f67-8746-f3e8b192d286 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:13.813535505 +0000 UTC m=+41.671224824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-d86sm" (UID: "1e94ffbd-080c-4f67-8746-f3e8b192d286") : secret "prometheus-operator-tls" not found Sep 07 10:21:13.314367 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.314342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e94ffbd-080c-4f67-8746-f3e8b192d286-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.317394 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.317372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.323861 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.323837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2f4\" (UniqueName: \"kubernetes.io/projected/1e94ffbd-080c-4f67-8746-f3e8b192d286-kube-api-access-cr2f4\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.818707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.818662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:13.821762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:13.821731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e94ffbd-080c-4f67-8746-f3e8b192d286-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-d86sm\" (UID: \"1e94ffbd-080c-4f67-8746-f3e8b192d286\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:14.029710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.029667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" Sep 07 10:21:14.276041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.276004 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:21:14.297961 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.297919 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:21:14.298134 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.298026 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.300357 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300334 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 07 10:21:14.300489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300425 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 07 10:21:14.300489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300475 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 07 10:21:14.300697 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300674 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 07 10:21:14.300840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300700 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 07 10:21:14.300840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300779 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 07 10:21:14.300840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300827 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dflgf\"" Sep 07 10:21:14.301030 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.300977 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 07 10:21:14.424575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49tlc\" (UniqueName: \"kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.424789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424664 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.424789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.424789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.424911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.424911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.424823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.525790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.525758 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.525968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.525800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.525968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.525941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.525987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.526020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.526048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49tlc\" (UniqueName: \"kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526569 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.526541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526700 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.526616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.526756 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.526708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.528825 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.528798 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.529049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.529031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.535480 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.535459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49tlc\" (UniqueName: \"kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc\") pod \"console-78d796d844-fzgll\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:14.607579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:14.607536 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:15.242539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:15.242438 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-d86sm"] Sep 07 10:21:15.246270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:15.246241 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:21:15.275934 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:15.275899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e94ffbd_080c_4f67_8746_f3e8b192d286.slice/crio-517c0859b8126e0b495212ceffed8947842c4c6a46075dbf24fbefedcf42ebe0 WatchSource:0}: Error finding container 517c0859b8126e0b495212ceffed8947842c4c6a46075dbf24fbefedcf42ebe0: Status 404 returned error can't find the container with id 517c0859b8126e0b495212ceffed8947842c4c6a46075dbf24fbefedcf42ebe0 Sep 07 10:21:16.045701 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.045662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn7dn" event={"ID":"1c139569-281b-487b-9851-616f975116b1","Type":"ContainerStarted","Data":"7e54cf41287ec6f7ceae9beb997b3bd0bb846b339593013becf32dbd18c59782"} Sep 07 10:21:16.046195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.045766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fn7dn" event={"ID":"1c139569-281b-487b-9851-616f975116b1","Type":"ContainerStarted","Data":"94d4ca89055e742085f1df75bc8d0b829dca2dcca8368a9095b807fdc146a906"} Sep 07 10:21:16.046195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.045796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:16.048086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.048053 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xbrkj" event={"ID":"0b9a1cae-7394-4c14-a23b-4c1d35f5d258","Type":"ContainerStarted","Data":"ce9eb9621a29b05f009257644179912e3023d5ee6b2b003a5fa0f3061fb31daf"} Sep 07 10:21:16.049641 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.049611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cdcwk" event={"ID":"af0133f0-b811-47d0-b008-bab7a80516b9","Type":"ContainerStarted","Data":"c1c895451eb507c621c37872291ef3a0b5b16ecf6f3b1f47fb459d13abbdf7ac"} Sep 07 10:21:16.050790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.050770 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" event={"ID":"1e94ffbd-080c-4f67-8746-f3e8b192d286","Type":"ContainerStarted","Data":"517c0859b8126e0b495212ceffed8947842c4c6a46075dbf24fbefedcf42ebe0"} Sep 07 10:21:16.051800 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.051782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d796d844-fzgll" event={"ID":"394eedcf-32dd-49e6-a8d1-a873663aed03","Type":"ContainerStarted","Data":"914cc097afc37e9806591b01ea51f84437cc2e6d7aae5664f23d72bbe37f3f32"} Sep 07 10:21:16.061351 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.061301 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fn7dn" podStartSLOduration=2.302122865 podStartE2EDuration="6.061282589s" totalImagesPullingTime="3.759159724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:16.061104767 +0000 UTC m=+43.918794087" watchObservedRunningTime="2026-09-07 10:21:16.061282589 +0000 UTC m=+43.918971902" Sep 07 10:21:16.074987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:16.074926 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cdcwk" podStartSLOduration=2.249038341 podStartE2EDuration="6.074909171s" totalImagesPullingTime="3.82587083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:16.074017204 +0000 UTC m=+43.931706547" watchObservedRunningTime="2026-09-07 10:21:16.074909171 +0000 UTC m=+43.932598487" Sep 07 10:21:18.064437 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:18.064397 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" event={"ID":"1e94ffbd-080c-4f67-8746-f3e8b192d286","Type":"ContainerStarted","Data":"bc9bed091f91f1c68cfdd54152a9d18f13ebf8126b65b9059662e4f5fc6641df"} Sep 07 10:21:18.064988 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:18.064463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" event={"ID":"1e94ffbd-080c-4f67-8746-f3e8b192d286","Type":"ContainerStarted","Data":"aa79c637895e818b459a678e10c4fedde6e81f6e52941f411287592de989a754"} Sep 07 10:21:18.081653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:18.081578 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xbrkj" podStartSLOduration=4.219457766 podStartE2EDuration="8.08155468s" totalImagesPullingTime="3.862096914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:21:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:16.092782785 +0000 UTC m=+43.950472092" watchObservedRunningTime="2026-09-07 10:21:18.08155468 +0000 UTC m=+45.939243997" Sep 07 10:21:18.081854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:18.081732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-d86sm" podStartSLOduration=3.30158633 podStartE2EDuration="5.081722499s" totalImagesPullingTime="1.780136169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:18.081309365 +0000 UTC m=+45.938998684" watchObservedRunningTime="2026-09-07 10:21:18.081722499 +0000 UTC m=+45.939411816" Sep 07 10:21:19.069360 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:19.069319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d796d844-fzgll" event={"ID":"394eedcf-32dd-49e6-a8d1-a873663aed03","Type":"ContainerStarted","Data":"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6"} Sep 07 10:21:19.085199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:19.085141 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78d796d844-fzgll" podStartSLOduration=1.5171306900000001 podStartE2EDuration="5.085124631s" totalImagesPullingTime="3.567993941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:19.084749115 +0000 UTC m=+46.942438427" watchObservedRunningTime="2026-09-07 10:21:19.085124631 +0000 UTC m=+46.942814052" Sep 07 10:21:20.459804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.459766 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fs698"] Sep 07 10:21:20.475359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.475318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.479954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.479928 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-v4m4d\"" Sep 07 10:21:20.480463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.480446 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 07 10:21:20.481937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.481817 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 07 10:21:20.486888 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.486860 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fs698"] Sep 07 10:21:20.515604 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.515551 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8"] Sep 07 10:21:20.537149 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.537117 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4k6fx"] Sep 07 10:21:20.537322 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.537258 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.539479 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.539453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sbxqg\"" Sep 07 10:21:20.539642 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.539543 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 07 10:21:20.539721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.539692 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 07 10:21:20.539778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.539757 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 07 10:21:20.556972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.556935 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8"] Sep 07 10:21:20.557128 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.557057 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.559009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.558986 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 07 10:21:20.559150 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.559102 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 07 10:21:20.559150 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.559132 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 07 10:21:20.559265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.559225 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8f4ct\"" Sep 07 10:21:20.576900 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.576870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-metrics-client-ca\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.576908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-tls\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.576975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577043 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.577163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6d78436-0df6-4525-a79b-dfe5a2814eb5-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-wtmp\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577176 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-textfile\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-sys\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp2g9\" (UniqueName: \"kubernetes.io/projected/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-api-access-tp2g9\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.577409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577333 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.577409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577352 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7gv\" (UniqueName: \"kubernetes.io/projected/33e8e89f-9489-46e7-b15f-1c9b5f33be15-kube-api-access-kf7gv\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.577409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-root\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.577554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e8e89f-9489-46e7-b15f-1c9b5f33be15-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.577554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.577441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skh7p\" (UniqueName: \"kubernetes.io/projected/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-kube-api-access-skh7p\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.678711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.678711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6d78436-0df6-4525-a79b-dfe5a2814eb5-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678802 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-wtmp\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-textfile\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.678945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.678893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-wtmp\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-sys\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp2g9\" (UniqueName: \"kubernetes.io/projected/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-api-access-tp2g9\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.679242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7gv\" (UniqueName: \"kubernetes.io/projected/33e8e89f-9489-46e7-b15f-1c9b5f33be15-kube-api-access-kf7gv\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-textfile\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-root\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e8e89f-9489-46e7-b15f-1c9b5f33be15-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e6d78436-0df6-4525-a79b-dfe5a2814eb5-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skh7p\" (UniqueName: \"kubernetes.io/projected/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-kube-api-access-skh7p\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-metrics-client-ca\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.679610 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-tls\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.680014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.680014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.680014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.679884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-root\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.680695 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.680519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-sys\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.680695 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.680566 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-metrics-client-ca\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.680695 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:20.680577 2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 07 10:21:20.680971 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.680938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-accelerators-collector-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.681168 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.681060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/33e8e89f-9489-46e7-b15f-1c9b5f33be15-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.681284 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:20.681237 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls podName:e6d78436-0df6-4525-a79b-dfe5a2814eb5 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:21.181213163 +0000 UTC m=+49.038902466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-7ppg8" (UID: "e6d78436-0df6-4525-a79b-dfe5a2814eb5") : secret "kube-state-metrics-tls" not found Sep 07 10:21:20.682329 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.682307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.682489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.682469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/33e8e89f-9489-46e7-b15f-1c9b5f33be15-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.682568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.682547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.682709 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.682690 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-node-exporter-tls\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.682899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.682883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.688574 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.688549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skh7p\" (UniqueName: \"kubernetes.io/projected/ca7a9454-5371-4477-8efd-2b6ddf3c2fa2-kube-api-access-skh7p\") pod \"node-exporter-4k6fx\" (UID: \"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2\") " pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.688972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.688946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7gv\" (UniqueName: \"kubernetes.io/projected/33e8e89f-9489-46e7-b15f-1c9b5f33be15-kube-api-access-kf7gv\") pod \"openshift-state-metrics-657fd6c54-fs698\" (UID: \"33e8e89f-9489-46e7-b15f-1c9b5f33be15\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.691942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.691920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp2g9\" (UniqueName: \"kubernetes.io/projected/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-api-access-tp2g9\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:20.784688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.784648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" Sep 07 10:21:20.865917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:20.865874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4k6fx" Sep 07 10:21:20.909346 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:20.909311 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7a9454_5371_4477_8efd_2b6ddf3c2fa2.slice/crio-00a3a5f2782b9a16605576764c2a46501587e6cb75309948327ef7987f7bdb58 WatchSource:0}: Error finding container 00a3a5f2782b9a16605576764c2a46501587e6cb75309948327ef7987f7bdb58: Status 404 returned error can't find the container with id 00a3a5f2782b9a16605576764c2a46501587e6cb75309948327ef7987f7bdb58 Sep 07 10:21:21.078150 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.078062 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k6fx" event={"ID":"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2","Type":"ContainerStarted","Data":"00a3a5f2782b9a16605576764c2a46501587e6cb75309948327ef7987f7bdb58"} Sep 07 10:21:21.088557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.088529 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-fs698"] Sep 07 10:21:21.091776 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:21.091753 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e8e89f_9489_46e7_b15f_1c9b5f33be15.slice/crio-5a0089fcac034a6c0b5d877d20427a3bf99a45a119563989a581df7521a66b05 WatchSource:0}: Error finding container 5a0089fcac034a6c0b5d877d20427a3bf99a45a119563989a581df7521a66b05: Status 404 returned error can't find the container with id 5a0089fcac034a6c0b5d877d20427a3bf99a45a119563989a581df7521a66b05 Sep 07 10:21:21.184686 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.184651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:21.187433 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.187407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6d78436-0df6-4525-a79b-dfe5a2814eb5-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-7ppg8\" (UID: \"e6d78436-0df6-4525-a79b-dfe5a2814eb5\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:21.446762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.446726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" Sep 07 10:21:21.538087 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.538057 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:21:21.546842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.546807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.548816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.548786 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 07 10:21:21.549179 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549159 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hjpgr\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549517 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549558 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549608 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549608 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549623 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549662 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549776 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 07 10:21:21.550061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.549955 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 07 10:21:21.558685 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.558660 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:21:21.586987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.586952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.586987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.586985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587263 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpg2x\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.587541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.587348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688249 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpg2x\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:21.688392 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle podName:c0fe034b-281f-4ef6-bde2-67dd927f653c nodeName:}" failed. No retries permitted until 2026-09-07 10:21:22.188367247 +0000 UTC m=+50.046056569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c") : configmap references non-existent config key: ca-bundle.crt Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.688968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.688676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.691838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.691725 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.691838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.691785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.692043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.691725 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.692135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.692100 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.692135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.692118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.692308 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.692254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.692561 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.692516 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.693186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.693161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.693286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.693186 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.694613 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.694570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.695739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.695722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpg2x\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:21.733224 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:21.733190 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8"] Sep 07 10:21:21.736180 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:21.736150 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d78436_0df6_4525_a79b_dfe5a2814eb5.slice/crio-090a84a8e3ca906d27301a5b90e382fe789adb52dba7ee29b397e9a57ad221fd WatchSource:0}: Error finding container 090a84a8e3ca906d27301a5b90e382fe789adb52dba7ee29b397e9a57ad221fd: Status 404 returned error can't find the container with id 090a84a8e3ca906d27301a5b90e382fe789adb52dba7ee29b397e9a57ad221fd Sep 07 10:21:22.083820 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.083776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" event={"ID":"e6d78436-0df6-4525-a79b-dfe5a2814eb5","Type":"ContainerStarted","Data":"090a84a8e3ca906d27301a5b90e382fe789adb52dba7ee29b397e9a57ad221fd"} Sep 07 10:21:22.086362 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.086327 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" event={"ID":"33e8e89f-9489-46e7-b15f-1c9b5f33be15","Type":"ContainerStarted","Data":"00c5ef48b23eaed5ef05b805b8114d68300752a61031e6a401480289d411467a"} Sep 07 10:21:22.086501 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.086408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" event={"ID":"33e8e89f-9489-46e7-b15f-1c9b5f33be15","Type":"ContainerStarted","Data":"65f7a3b5f7650647fe2e843bd07d48233eb5572ab47c433b3f75a9aed40f095f"} Sep 07 10:21:22.086552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.086504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" event={"ID":"33e8e89f-9489-46e7-b15f-1c9b5f33be15","Type":"ContainerStarted","Data":"5a0089fcac034a6c0b5d877d20427a3bf99a45a119563989a581df7521a66b05"} Sep 07 10:21:22.195127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.195092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:22.195800 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.195779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:22.453535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.453462 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-56775896b9-d2rjp"] Sep 07 10:21:22.462411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.462380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:21:22.466986 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.466822 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.468537 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.468513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56775896b9-d2rjp"] Sep 07 10:21:22.469223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469198 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8o1pijbaj3und\"" Sep 07 10:21:22.469344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469247 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 07 10:21:22.469344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 07 10:21:22.469344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469213 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 07 10:21:22.469487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 07 10:21:22.469487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x5b2h\"" Sep 07 10:21:22.469487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.469258 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 07 10:21:22.497764 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-metrics-client-ca\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.497764 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497764 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-grpc-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.497950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.497950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497883 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.497950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjz58\" (UniqueName: \"kubernetes.io/projected/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-kube-api-access-vjz58\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.498065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.497969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.498065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.498005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.498065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.498025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.598729 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-metrics-client-ca\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.598729 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-grpc-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjz58\" (UniqueName: \"kubernetes.io/projected/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-kube-api-access-vjz58\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.598984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.599037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.599069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.599502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.599436 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-metrics-client-ca\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.601769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.601731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.601769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.601731 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.601951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.601933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.601988 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.601948 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-grpc-tls\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.602079 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.602061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.602270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.602250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.607191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.607165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjz58\" (UniqueName: \"kubernetes.io/projected/fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7-kube-api-access-vjz58\") pod \"thanos-querier-56775896b9-d2rjp\" (UID: \"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7\") " pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:22.777952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:22.777897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:23.094211 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.094180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k6fx" event={"ID":"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2","Type":"ContainerStarted","Data":"ae853c0a199626f7603abeeaf5e4881e8881d3ddcb2adfa5f8ab78ba46b643b0"} Sep 07 10:21:23.096886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.096852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" event={"ID":"33e8e89f-9489-46e7-b15f-1c9b5f33be15","Type":"ContainerStarted","Data":"40b30406c4a537e491ac364fbe4728e37ad1cee4bb61e7528efea983367d3edc"} Sep 07 10:21:23.102733 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.102708 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:21:23.107907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.107878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-56775896b9-d2rjp"] Sep 07 10:21:23.129314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.129243 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-fs698" podStartSLOduration=1.6527471820000001 podStartE2EDuration="3.129220255s" totalImagesPullingTime="1.476473073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:23.128425195 +0000 UTC m=+50.986114512" watchObservedRunningTime="2026-09-07 10:21:23.129220255 +0000 UTC m=+50.986909572" Sep 07 10:21:23.135672 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:23.135631 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8656cf_c8d0_4010_b6f7_8bd8fccc53b7.slice/crio-d8ecd7397b4a2b1561f428de70aa6e5779647ecb523cd134a666cf4d0f3b54b1 WatchSource:0}: Error finding container d8ecd7397b4a2b1561f428de70aa6e5779647ecb523cd134a666cf4d0f3b54b1: Status 404 returned error can't find the container with id d8ecd7397b4a2b1561f428de70aa6e5779647ecb523cd134a666cf4d0f3b54b1 Sep 07 10:21:23.135918 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:23.135891 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fe034b_281f_4ef6_bde2_67dd927f653c.slice/crio-c08986da7922d2e841c914c947058830d50c192cec3a8f20296fdfe787f07804 WatchSource:0}: Error finding container c08986da7922d2e841c914c947058830d50c192cec3a8f20296fdfe787f07804: Status 404 returned error can't find the container with id c08986da7922d2e841c914c947058830d50c192cec3a8f20296fdfe787f07804 Sep 07 10:21:23.249928 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.249893 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:23.272565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.272533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.282114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.282075 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:23.284553 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.284530 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 07 10:21:23.306750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.306750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gtr\" (UniqueName: \"kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.307038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.307038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.307038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.307038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.306982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.307202 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.307040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408576 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gtr\" (UniqueName: \"kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.408928 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.408790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.409786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.409351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.409786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.409480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.409786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.409732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.410013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.409918 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.411792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.411772 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.412300 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.412277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.417670 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.417645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gtr\" (UniqueName: \"kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr\") pod \"console-5ff68658bb-bffkd\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.583989 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.583928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:23.894153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:23.894119 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:24.001800 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:24.001761 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08228660_eb6b_4682_a4da_c9ad862f71db.slice/crio-646af07b417a02dc004ad0eac64e4260845a0897d6f8613ee35428ac9cf6dae0 WatchSource:0}: Error finding container 646af07b417a02dc004ad0eac64e4260845a0897d6f8613ee35428ac9cf6dae0: Status 404 returned error can't find the container with id 646af07b417a02dc004ad0eac64e4260845a0897d6f8613ee35428ac9cf6dae0 Sep 07 10:21:24.103025 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.102930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"d8ecd7397b4a2b1561f428de70aa6e5779647ecb523cd134a666cf4d0f3b54b1"} Sep 07 10:21:24.104651 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.104619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff68658bb-bffkd" event={"ID":"08228660-eb6b-4682-a4da-c9ad862f71db","Type":"ContainerStarted","Data":"646af07b417a02dc004ad0eac64e4260845a0897d6f8613ee35428ac9cf6dae0"} Sep 07 10:21:24.107495 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.107139 2577 generic.go:356] "Generic (PLEG): container finished" podID="ca7a9454-5371-4477-8efd-2b6ddf3c2fa2" containerID="ae853c0a199626f7603abeeaf5e4881e8881d3ddcb2adfa5f8ab78ba46b643b0" exitCode=0 Sep 07 10:21:24.107495 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.107241 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k6fx" event={"ID":"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2","Type":"ContainerDied","Data":"ae853c0a199626f7603abeeaf5e4881e8881d3ddcb2adfa5f8ab78ba46b643b0"} Sep 07 10:21:24.110101 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.110026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"c08986da7922d2e841c914c947058830d50c192cec3a8f20296fdfe787f07804"} Sep 07 10:21:24.608075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.607982 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:24.608075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.608063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:24.615191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.615162 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:24.780679 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.780643 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5559b5685-dbj7z"] Sep 07 10:21:24.796716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.796693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5559b5685-dbj7z"] Sep 07 10:21:24.797727 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.796794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.799269 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 07 10:21:24.799269 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bhvcm\"" Sep 07 10:21:24.799269 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799164 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 07 10:21:24.799501 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799325 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b41qfbfvjpad5\"" Sep 07 10:21:24.799501 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799328 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 07 10:21:24.799501 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.799494 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 07 10:21:24.825716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-tls\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.825848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-client-certs\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.825848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-metrics-server-audit-profiles\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.825930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77lr\" (UniqueName: \"kubernetes.io/projected/e9944e5c-b7cc-4865-abee-cc5393a49024-kube-api-access-q77lr\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.825930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9944e5c-b7cc-4865-abee-cc5393a49024-audit-log\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.826006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.825927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.826043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.826018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-client-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.926974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-tls\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927035 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-client-certs\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-metrics-server-audit-profiles\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77lr\" (UniqueName: \"kubernetes.io/projected/e9944e5c-b7cc-4865-abee-cc5393a49024-kube-api-access-q77lr\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9944e5c-b7cc-4865-abee-cc5393a49024-audit-log\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-client-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.927846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.927818 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e9944e5c-b7cc-4865-abee-cc5393a49024-audit-log\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.928600 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.928562 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.928703 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.928663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e9944e5c-b7cc-4865-abee-cc5393a49024-metrics-server-audit-profiles\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.930052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.930013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-client-certs\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.930131 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.930051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-secret-metrics-server-tls\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.931098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.931079 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e9944e5c-b7cc-4865-abee-cc5393a49024-client-ca-bundle\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:24.935375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:24.935353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77lr\" (UniqueName: \"kubernetes.io/projected/e9944e5c-b7cc-4865-abee-cc5393a49024-kube-api-access-q77lr\") pod \"metrics-server-5559b5685-dbj7z\" (UID: \"e9944e5c-b7cc-4865-abee-cc5393a49024\") " pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:25.115667 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.115549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff68658bb-bffkd" event={"ID":"08228660-eb6b-4682-a4da-c9ad862f71db","Type":"ContainerStarted","Data":"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3"} Sep 07 10:21:25.118061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.118024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k6fx" event={"ID":"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2","Type":"ContainerStarted","Data":"397df74e120ee45d465b9b7a5d0aa2debf0b35c3d6d4285b4112e6b5b1b848eb"} Sep 07 10:21:25.118238 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.118087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4k6fx" event={"ID":"ca7a9454-5371-4477-8efd-2b6ddf3c2fa2","Type":"ContainerStarted","Data":"bd71e4f22db27c45c83a6ecf3825bd5e2c7d7796edf15f2cbace027ddb2a59df"} Sep 07 10:21:25.124052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.124017 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:25.125649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.125619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" event={"ID":"e6d78436-0df6-4525-a79b-dfe5a2814eb5","Type":"ContainerStarted","Data":"a121dc806ec6bc11ed2dcbf1eb87d936452bde95dcb50ff690a2b3ea7c5d6b06"} Sep 07 10:21:25.125773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.125708 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" event={"ID":"e6d78436-0df6-4525-a79b-dfe5a2814eb5","Type":"ContainerStarted","Data":"559da283087c28d143559d31b84edd0369911bfe5f7cbfde74f52bcca55ede73"} Sep 07 10:21:25.125773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.125739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" event={"ID":"e6d78436-0df6-4525-a79b-dfe5a2814eb5","Type":"ContainerStarted","Data":"c9c7980d980339fe7810c6c05d178894b03cb071c600c619c7ab8e627f2f44fb"} Sep 07 10:21:25.127495 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.127471 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5" exitCode=0 Sep 07 10:21:25.127604 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.127540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5"} Sep 07 10:21:25.131457 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.131409 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ff68658bb-bffkd" podStartSLOduration=2.13138971 podStartE2EDuration="2.13138971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:25.13082027 +0000 UTC m=+52.988509587" watchObservedRunningTime="2026-09-07 10:21:25.13138971 +0000 UTC m=+52.989079027" Sep 07 10:21:25.132520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.132497 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:21:25.147951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.147883 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-7ppg8" podStartSLOduration=2.858688141 podStartE2EDuration="5.147858981s" totalImagesPullingTime="2.28917084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:25.146183924 +0000 UTC m=+53.003873240" watchObservedRunningTime="2026-09-07 10:21:25.147858981 +0000 UTC m=+53.005548299" Sep 07 10:21:25.188577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.188522 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4k6fx" podStartSLOduration=3.174568627 podStartE2EDuration="5.188505773s" totalImagesPullingTime="1.909015937s" totalInitContainerRuntime="104.921209ms" isStatefulPod=true podCreationTimestamp="2026-09-07 10:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:25.187522372 +0000 UTC m=+53.045211691" watchObservedRunningTime="2026-09-07 10:21:25.188505773 +0000 UTC m=+53.046195099" Sep 07 10:21:25.251915 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.251398 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5"] Sep 07 10:21:25.254090 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.254048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.256167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.256133 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 07 10:21:25.257286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.256393 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jzczx\"" Sep 07 10:21:25.262495 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.262453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5"] Sep 07 10:21:25.331147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.331100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nb67\" (UniqueName: \"kubernetes.io/projected/0bec0a65-32ef-41e0-8931-7c46306d0139-kube-api-access-5nb67\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.331327 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.331165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.432265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.432171 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nb67\" (UniqueName: \"kubernetes.io/projected/0bec0a65-32ef-41e0-8931-7c46306d0139-kube-api-access-5nb67\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.432265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.432235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.432477 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:25.432367 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 07 10:21:25.432477 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:25.432438 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert podName:0bec0a65-32ef-41e0-8931-7c46306d0139 nodeName:}" failed. No retries permitted until 2026-09-07 10:21:25.932417877 +0000 UTC m=+53.790107194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-8j2z5" (UID: "0bec0a65-32ef-41e0-8931-7c46306d0139") : secret "monitoring-plugin-cert" not found Sep 07 10:21:25.442347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.442316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nb67\" (UniqueName: \"kubernetes.io/projected/0bec0a65-32ef-41e0-8931-7c46306d0139-kube-api-access-5nb67\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.704619 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.703079 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78c745d5f8-45r9t"] Sep 07 10:21:25.711766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.711743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.714108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.714075 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 07 10:21:25.714225 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.714120 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 07 10:21:25.714395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.714375 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 07 10:21:25.714563 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.714541 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-m5xtj\"" Sep 07 10:21:25.714892 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.714479 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 07 10:21:25.715431 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.715404 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 07 10:21:25.721380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.721155 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 07 10:21:25.724330 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.724306 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78c745d5f8-45r9t"] Sep 07 10:21:25.735568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735303 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-serving-certs-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-federate-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735974 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735974 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-metrics-client-ca\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.735974 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.735895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbqh\" (UniqueName: \"kubernetes.io/projected/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-kube-api-access-6sbqh\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.836948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.836912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-serving-certs-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.836954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.836993 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-federate-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837302 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-metrics-client-ca\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837302 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbqh\" (UniqueName: \"kubernetes.io/projected/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-kube-api-access-6sbqh\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837302 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.837913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.837886 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-serving-certs-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.838440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.838411 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.839110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.839081 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-metrics-client-ca\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.840818 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.840773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-federate-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.841080 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.841023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.841528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.841509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-secret-telemeter-client\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.841625 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.841509 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-telemeter-client-tls\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.849484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.849459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbqh\" (UniqueName: \"kubernetes.io/projected/b9bc4b8e-17a5-4649-b4f0-729d3ec3c199-kube-api-access-6sbqh\") pod \"telemeter-client-78c745d5f8-45r9t\" (UID: \"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199\") " pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:25.930390 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.930353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5559b5685-dbj7z"] Sep 07 10:21:25.934542 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:25.934479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9944e5c_b7cc_4865_abee_cc5393a49024.slice/crio-f3889da78232a618de1a2b7f6d6213548c93bf374b568bea766581134619c761 WatchSource:0}: Error finding container f3889da78232a618de1a2b7f6d6213548c93bf374b568bea766581134619c761: Status 404 returned error can't find the container with id f3889da78232a618de1a2b7f6d6213548c93bf374b568bea766581134619c761 Sep 07 10:21:25.938712 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.938675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:25.941573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:25.941548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bec0a65-32ef-41e0-8931-7c46306d0139-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-8j2z5\" (UID: \"0bec0a65-32ef-41e0-8931-7c46306d0139\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:26.035493 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.035455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" Sep 07 10:21:26.060329 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.060303 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fn7dn" Sep 07 10:21:26.135965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.135926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"bdf9c2bc8ec8ff23366b8385a4c3e48d16e7cd17d3f3ef2c64aa014ce8ce296b"} Sep 07 10:21:26.136138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.136034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"c9642e0f4dbcbbed3f09ac7733a85d92cd60119c5f7a922acb449a536d3cb1b1"} Sep 07 10:21:26.136138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.136063 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"f49b1e2ea627affcfbce1444d42c565f2eccd77a36facc4a26ea246589e51a8d"} Sep 07 10:21:26.138274 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.138249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" event={"ID":"e9944e5c-b7cc-4865-abee-cc5393a49024","Type":"ContainerStarted","Data":"f3889da78232a618de1a2b7f6d6213548c93bf374b568bea766581134619c761"} Sep 07 10:21:26.167412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.167379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:26.314043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.314010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78c745d5f8-45r9t"] Sep 07 10:21:26.317325 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:26.317296 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bc4b8e_17a5_4649_b4f0_729d3ec3c199.slice/crio-c6add7cc29484e69649b97aee0dba8120eb394a2653880c7949ce27c473b7dd9 WatchSource:0}: Error finding container c6add7cc29484e69649b97aee0dba8120eb394a2653880c7949ce27c473b7dd9: Status 404 returned error can't find the container with id c6add7cc29484e69649b97aee0dba8120eb394a2653880c7949ce27c473b7dd9 Sep 07 10:21:26.441994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.441956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5"] Sep 07 10:21:26.448158 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:26.448125 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bec0a65_32ef_41e0_8931_7c46306d0139.slice/crio-541f7a6057af7b17c0fc1442f2231e6804ff7edd5f7252248f8f02c56bf8414b WatchSource:0}: Error finding container 541f7a6057af7b17c0fc1442f2231e6804ff7edd5f7252248f8f02c56bf8414b: Status 404 returned error can't find the container with id 541f7a6057af7b17c0fc1442f2231e6804ff7edd5f7252248f8f02c56bf8414b Sep 07 10:21:26.763652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.758383 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:21:26.763652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.763559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771047 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771109 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771309 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771380 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dlam26pi8q80t\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771389 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771640 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771778 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771849 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.771993 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 07 10:21:26.772160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.772008 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 07 10:21:26.772804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.772243 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 07 10:21:26.772804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.772432 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6mgl\"" Sep 07 10:21:26.772804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.772609 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 07 10:21:26.782942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.780901 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 07 10:21:26.782942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.782392 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 07 10:21:26.788892 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.788855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:21:26.849979 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.849937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.849987 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850047 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8kz\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850331 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850556 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.850899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.850604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8kz\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.952477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.953808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.953808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.953808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.951951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.953808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.952869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.957133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.956144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.957133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.956555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.957133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.956857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.958766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.957752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.958766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.958286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.958766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.958453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.958766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.958732 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.959978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.959521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.959978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.959925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.960343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.960315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.962421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.962397 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.963333 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.963284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.965233 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.965185 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.965884 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.965814 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.965884 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.965837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.966019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.965925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.966702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.966678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.967276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.967250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8kz\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:26.969186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:26.969153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:27.090315 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:27.089924 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:27.154286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:27.154230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" event={"ID":"0bec0a65-32ef-41e0-8931-7c46306d0139","Type":"ContainerStarted","Data":"541f7a6057af7b17c0fc1442f2231e6804ff7edd5f7252248f8f02c56bf8414b"} Sep 07 10:21:27.156390 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:27.156319 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" event={"ID":"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199","Type":"ContainerStarted","Data":"c6add7cc29484e69649b97aee0dba8120eb394a2653880c7949ce27c473b7dd9"} Sep 07 10:21:28.672667 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:28.672568 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:21:28.740435 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:28.740398 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0e7d70_eea4_4dca_9b87_3fb05663c60c.slice/crio-a95b0326bc74043a1516c774f4cfdbda1c29f9c6ce077b410f7c0303efc55982 WatchSource:0}: Error finding container a95b0326bc74043a1516c774f4cfdbda1c29f9c6ce077b410f7c0303efc55982: Status 404 returned error can't find the container with id a95b0326bc74043a1516c774f4cfdbda1c29f9c6ce077b410f7c0303efc55982 Sep 07 10:21:29.168049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.168014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"e85400b4f54f194e414efe924f97422b3203a17cac23d0c7fa51e8d2799c091d"} Sep 07 10:21:29.168209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.168085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"15fb6c7ef1f439113e814979696e3cfcaef22c63da77d3ada42e35c11f9136e4"} Sep 07 10:21:29.168209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.168114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" event={"ID":"fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7","Type":"ContainerStarted","Data":"57018f6272cca07703b071e48617b5c2a29982a5d43894c5c77a9f5733601b11"} Sep 07 10:21:29.168815 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.168762 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:29.169969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.169947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" event={"ID":"e9944e5c-b7cc-4865-abee-cc5393a49024","Type":"ContainerStarted","Data":"62edbfc3cdeae2acfa5318e9bf18f2bc6daf2cb84669107d442ce9aae7aced94"} Sep 07 10:21:29.171089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.171061 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" event={"ID":"0bec0a65-32ef-41e0-8931-7c46306d0139","Type":"ContainerStarted","Data":"1d8362601d264743063ac27d5e50fdf53fcacff1df6ebea2ac0f541097d8ecf7"} Sep 07 10:21:29.171331 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.171293 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:29.172924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.172646 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" exitCode=0 Sep 07 10:21:29.172924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.172771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} Sep 07 10:21:29.172924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.172814 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"a95b0326bc74043a1516c774f4cfdbda1c29f9c6ce077b410f7c0303efc55982"} Sep 07 10:21:29.177162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.177015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" Sep 07 10:21:29.177162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.177059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1"} Sep 07 10:21:29.177162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.177108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a"} Sep 07 10:21:29.177162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.177134 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5"} Sep 07 10:21:29.179652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.179628 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" event={"ID":"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199","Type":"ContainerStarted","Data":"d0061f662555760be28442ea05afd53538f36607ad370988f290ee8943b69fea"} Sep 07 10:21:29.179746 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.179676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" event={"ID":"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199","Type":"ContainerStarted","Data":"5d5037254d91c33a76215d543d4ed3f833e649908aeea990440ec7d98b100867"} Sep 07 10:21:29.179746 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.179698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" event={"ID":"b9bc4b8e-17a5-4649-b4f0-729d3ec3c199","Type":"ContainerStarted","Data":"7e6bb94ade50f8c59183c7f688dd1a1c5912f1846280ccb0c5073965ee607909"} Sep 07 10:21:29.190403 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.190169 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" podStartSLOduration=1.804362845 podStartE2EDuration="7.190151127s" totalImagesPullingTime="5.385788282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:29.187950625 +0000 UTC m=+57.045639944" watchObservedRunningTime="2026-09-07 10:21:29.190151127 +0000 UTC m=+57.047840446" Sep 07 10:21:29.204925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.204856 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" podStartSLOduration=2.403268418 podStartE2EDuration="5.204834917s" totalImagesPullingTime="2.801566499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:29.202818922 +0000 UTC m=+57.060508240" watchObservedRunningTime="2026-09-07 10:21:29.204834917 +0000 UTC m=+57.062524236" Sep 07 10:21:29.227123 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.226786 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78c745d5f8-45r9t" podStartSLOduration=1.787098203 podStartE2EDuration="4.226767081s" totalImagesPullingTime="2.439668878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:29.225643223 +0000 UTC m=+57.083332540" watchObservedRunningTime="2026-09-07 10:21:29.226767081 +0000 UTC m=+57.084456396" Sep 07 10:21:29.240402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.240344 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-8j2z5" podStartSLOduration=1.931192374 podStartE2EDuration="4.240322589s" totalImagesPullingTime="2.309130215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:29.23980665 +0000 UTC m=+57.097495970" watchObservedRunningTime="2026-09-07 10:21:29.240322589 +0000 UTC m=+57.098011901" Sep 07 10:21:29.472114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:29.472076 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:30.190191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:30.190021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83"} Sep 07 10:21:30.190191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:30.190113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc"} Sep 07 10:21:30.190191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:30.190152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerStarted","Data":"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103"} Sep 07 10:21:30.199624 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:30.199574 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-56775896b9-d2rjp" Sep 07 10:21:30.215937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:30.215873 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.862364213 podStartE2EDuration="9.215850001s" totalImagesPullingTime="5.334985542s" totalInitContainerRuntime="18.500246ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:30.214236238 +0000 UTC m=+58.071925555" watchObservedRunningTime="2026-09-07 10:21:30.215850001 +0000 UTC m=+58.073539318" Sep 07 10:21:32.001215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:32.001181 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qpbw" Sep 07 10:21:33.207205 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} Sep 07 10:21:33.207205 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} Sep 07 10:21:33.207657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} Sep 07 10:21:33.207657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} Sep 07 10:21:33.207657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207261 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} Sep 07 10:21:33.207657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.207283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerStarted","Data":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} Sep 07 10:21:33.233790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.233737 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.730602197 podStartE2EDuration="7.233718952s" totalImagesPullingTime="3.462720804s" totalInitContainerRuntime="40.395951ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:33.231814827 +0000 UTC m=+61.089504139" watchObservedRunningTime="2026-09-07 10:21:33.233718952 +0000 UTC m=+61.091408268" Sep 07 10:21:33.584803 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:33.584752 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:37.090775 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:37.090735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:38.389960 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.389911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:38.392149 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.392127 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 07 10:21:38.403316 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.403288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b2886c9-c467-41ea-aa26-db0748ea4400-metrics-certs\") pod \"network-metrics-daemon-z5f5r\" (UID: \"4b2886c9-c467-41ea-aa26-db0748ea4400\") " pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:38.437005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.436968 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbx74\"" Sep 07 10:21:38.445062 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.445029 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-z5f5r" Sep 07 10:21:38.490648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.490571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:38.493077 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.493053 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 07 10:21:38.503666 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.503634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb112766-fde8-43ff-9a61-36845bf0f30f-original-pull-secret\") pod \"global-pull-secret-syncer-b5fbr\" (UID: \"eb112766-fde8-43ff-9a61-36845bf0f30f\") " pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:38.591738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.591699 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:38.594161 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.594144 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 07 10:21:38.604560 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.604529 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 07 10:21:38.615903 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.615877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jlc6\" (UniqueName: \"kubernetes.io/projected/566e9ed2-77d3-4caf-82e5-5ceae47df598-kube-api-access-8jlc6\") pod \"network-check-target-97grq\" (UID: \"566e9ed2-77d3-4caf-82e5-5ceae47df598\") " pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:38.729520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.729493 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jrgm\"" Sep 07 10:21:38.738132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.738106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:38.740448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.740422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b5fbr" Sep 07 10:21:38.744028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:38.744000 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-z5f5r"] Sep 07 10:21:38.747024 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:38.746994 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2886c9_c467_41ea_aa26_db0748ea4400.slice/crio-38a25fa620fa3ce5818714b436e70c71d4b4597004fcff7bcb23caa1e37a128b WatchSource:0}: Error finding container 38a25fa620fa3ce5818714b436e70c71d4b4597004fcff7bcb23caa1e37a128b: Status 404 returned error can't find the container with id 38a25fa620fa3ce5818714b436e70c71d4b4597004fcff7bcb23caa1e37a128b Sep 07 10:21:39.037098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:39.036979 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-97grq"] Sep 07 10:21:39.039658 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:39.039633 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566e9ed2_77d3_4caf_82e5_5ceae47df598.slice/crio-2af5cc2432abb55a65e168e2da57b904789cce9ee06c8fb14b5f8f4c751e3b37 WatchSource:0}: Error finding container 2af5cc2432abb55a65e168e2da57b904789cce9ee06c8fb14b5f8f4c751e3b37: Status 404 returned error can't find the container with id 2af5cc2432abb55a65e168e2da57b904789cce9ee06c8fb14b5f8f4c751e3b37 Sep 07 10:21:39.061524 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:39.061491 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b5fbr"] Sep 07 10:21:39.064564 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:21:39.064537 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb112766_fde8_43ff_9a61_36845bf0f30f.slice/crio-0d380589b39e077758419a98baca64ef485482898daa867a169667c9b83d3a2b WatchSource:0}: Error finding container 0d380589b39e077758419a98baca64ef485482898daa867a169667c9b83d3a2b: Status 404 returned error can't find the container with id 0d380589b39e077758419a98baca64ef485482898daa867a169667c9b83d3a2b Sep 07 10:21:39.235806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:39.235766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5f5r" event={"ID":"4b2886c9-c467-41ea-aa26-db0748ea4400","Type":"ContainerStarted","Data":"38a25fa620fa3ce5818714b436e70c71d4b4597004fcff7bcb23caa1e37a128b"} Sep 07 10:21:39.237366 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:39.237323 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b5fbr" event={"ID":"eb112766-fde8-43ff-9a61-36845bf0f30f","Type":"ContainerStarted","Data":"0d380589b39e077758419a98baca64ef485482898daa867a169667c9b83d3a2b"} Sep 07 10:21:39.238793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:39.238772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-97grq" event={"ID":"566e9ed2-77d3-4caf-82e5-5ceae47df598","Type":"ContainerStarted","Data":"2af5cc2432abb55a65e168e2da57b904789cce9ee06c8fb14b5f8f4c751e3b37"} Sep 07 10:21:40.247407 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:40.247357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5f5r" event={"ID":"4b2886c9-c467-41ea-aa26-db0748ea4400","Type":"ContainerStarted","Data":"1aa0d8336bb8c3ee3da8e6f92e656bdd6abd12ecc859ddb370a39c1240a3377d"} Sep 07 10:21:40.247845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:40.247477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-z5f5r" event={"ID":"4b2886c9-c467-41ea-aa26-db0748ea4400","Type":"ContainerStarted","Data":"a6badba0d208d256d530da92a856bb5760a10e1b6ca80c495dfa47de93ea3a43"} Sep 07 10:21:40.264574 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:40.264517 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-z5f5r" podStartSLOduration=66.311475852 podStartE2EDuration="1m7.264492534s" totalImagesPullingTime="953.016682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:40.262857071 +0000 UTC m=+68.120546423" watchObservedRunningTime="2026-09-07 10:21:40.264492534 +0000 UTC m=+68.122181851" Sep 07 10:21:40.574281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:40.574197 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:21:44.272546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:44.272406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b5fbr" event={"ID":"eb112766-fde8-43ff-9a61-36845bf0f30f","Type":"ContainerStarted","Data":"a1520c9f1eccd740ec74ff6f2979543900f39e3c979a02d8b30b32ae0f81ef9e"} Sep 07 10:21:44.274499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:44.274464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-97grq" event={"ID":"566e9ed2-77d3-4caf-82e5-5ceae47df598","Type":"ContainerStarted","Data":"801e520e9ff9dbad1942a6976868e7f7a3ecd9fd7e6795efedd727dc505adc47"} Sep 07 10:21:44.274665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:44.274562 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:21:44.286537 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:44.286466 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b5fbr" podStartSLOduration=66.296947821 podStartE2EDuration="1m11.286443682s" totalImagesPullingTime="4.989495861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:44.28567992 +0000 UTC m=+72.143369237" watchObservedRunningTime="2026-09-07 10:21:44.286443682 +0000 UTC m=+72.144132998" Sep 07 10:21:44.299165 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:44.299104 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-97grq" podStartSLOduration=66.288291455 podStartE2EDuration="1m11.299083386s" totalImagesPullingTime="5.010791931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:21:44.298491919 +0000 UTC m=+72.156181235" watchObservedRunningTime="2026-09-07 10:21:44.299083386 +0000 UTC m=+72.156772703" Sep 07 10:21:45.124560 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:45.124517 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:45.124752 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:45.124615 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:21:46.484488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:46.484453 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:46.504421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:46.504392 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:47.107845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:47.107806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:21:54.492786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.492657 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ff68658bb-bffkd" podUID="08228660-eb6b-4682-a4da-c9ad862f71db" containerName="console" containerID="cri-o://352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3" gracePeriod=15 Sep 07 10:21:54.733822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.733795 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:54.849897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849797 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.849897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849907 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8gtr\" (UniqueName: \"kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.849986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.850037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert\") pod \"08228660-eb6b-4682-a4da-c9ad862f71db\" (UID: \"08228660-eb6b-4682-a4da-c9ad862f71db\") " Sep 07 10:21:54.850336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.850265 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca" (OuterVolumeSpecName: "service-ca") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:21:54.850395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.850337 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config" (OuterVolumeSpecName: "console-config") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:21:54.850523 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.850496 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:21:54.850523 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.850507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:21:54.852528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.852503 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:21:54.852665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.852528 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr" (OuterVolumeSpecName: "kube-api-access-l8gtr") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "kube-api-access-l8gtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:21:54.852665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.852569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "08228660-eb6b-4682-a4da-c9ad862f71db" (UID: "08228660-eb6b-4682-a4da-c9ad862f71db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:21:54.950917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950876 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.950917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950907 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-console-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.950917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950916 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gtr\" (UniqueName: \"kubernetes.io/projected/08228660-eb6b-4682-a4da-c9ad862f71db-kube-api-access-l8gtr\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.950917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950926 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-oauth-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.951188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950935 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/08228660-eb6b-4682-a4da-c9ad862f71db-console-serving-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.951188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950943 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-trusted-ca-bundle\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:54.951188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:54.950953 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/08228660-eb6b-4682-a4da-c9ad862f71db-oauth-serving-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:21:55.328136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.328098 2577 generic.go:356] "Generic (PLEG): container finished" podID="08228660-eb6b-4682-a4da-c9ad862f71db" containerID="352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3" exitCode=0 Sep 07 10:21:55.328336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.328143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ff68658bb-bffkd" Sep 07 10:21:55.328336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.328169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff68658bb-bffkd" event={"ID":"08228660-eb6b-4682-a4da-c9ad862f71db","Type":"ContainerDied","Data":"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3"} Sep 07 10:21:55.328336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.328220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ff68658bb-bffkd" event={"ID":"08228660-eb6b-4682-a4da-c9ad862f71db","Type":"ContainerDied","Data":"646af07b417a02dc004ad0eac64e4260845a0897d6f8613ee35428ac9cf6dae0"} Sep 07 10:21:55.328336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.328260 2577 scope.go:117] "RemoveContainer" containerID="352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3" Sep 07 10:21:55.338020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.338002 2577 scope.go:117] "RemoveContainer" containerID="352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3" Sep 07 10:21:55.338356 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:21:55.338326 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3\": container with ID starting with 352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3 not found: ID does not exist" containerID="352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3" Sep 07 10:21:55.338424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.338366 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3"} err="failed to get container status \"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3\": rpc error: code = NotFound desc = could not find container \"352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3\": container with ID starting with 352e6b788aed3717c9d907349d2cd38f768ad1ab24f16b421f329bd9ba3340a3 not found: ID does not exist" Sep 07 10:21:55.348813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.348786 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:55.351707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:55.351682 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ff68658bb-bffkd"] Sep 07 10:21:56.726811 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:21:56.726776 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08228660-eb6b-4682-a4da-c9ad862f71db" path="/var/lib/kubelet/pods/08228660-eb6b-4682-a4da-c9ad862f71db/volumes" Sep 07 10:22:05.129417 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.129372 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:22:05.133955 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.133928 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5559b5685-dbj7z" Sep 07 10:22:05.600886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.600817 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78d796d844-fzgll" podUID="394eedcf-32dd-49e6-a8d1-a873663aed03" containerName="console" containerID="cri-o://50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6" gracePeriod=15 Sep 07 10:22:05.849831 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.849803 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:22:05.945334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945234 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945274 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945306 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49tlc\" (UniqueName: \"kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945323 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945339 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945373 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca\") pod \"394eedcf-32dd-49e6-a8d1-a873663aed03\" (UID: \"394eedcf-32dd-49e6-a8d1-a873663aed03\") " Sep 07 10:22:05.945794 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945719 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config" (OuterVolumeSpecName: "console-config") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:05.945846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945790 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:05.945846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.945829 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca" (OuterVolumeSpecName: "service-ca") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:05.949813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.948307 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:05.949813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.948392 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc" (OuterVolumeSpecName: "kube-api-access-49tlc") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "kube-api-access-49tlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:22:05.949813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:05.948638 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "394eedcf-32dd-49e6-a8d1-a873663aed03" (UID: "394eedcf-32dd-49e6-a8d1-a873663aed03"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:06.046668 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046621 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-oauth-serving-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.046668 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046669 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.046920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046684 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-oauth-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.046920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046698 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394eedcf-32dd-49e6-a8d1-a873663aed03-console-serving-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.046920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046710 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49tlc\" (UniqueName: \"kubernetes.io/projected/394eedcf-32dd-49e6-a8d1-a873663aed03-kube-api-access-49tlc\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.046920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.046724 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394eedcf-32dd-49e6-a8d1-a873663aed03-console-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:06.375506 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.375469 2577 generic.go:356] "Generic (PLEG): container finished" podID="394eedcf-32dd-49e6-a8d1-a873663aed03" containerID="50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6" exitCode=0 Sep 07 10:22:06.375948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.375517 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78d796d844-fzgll" Sep 07 10:22:06.375948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.375558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d796d844-fzgll" event={"ID":"394eedcf-32dd-49e6-a8d1-a873663aed03","Type":"ContainerDied","Data":"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6"} Sep 07 10:22:06.375948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.375652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78d796d844-fzgll" event={"ID":"394eedcf-32dd-49e6-a8d1-a873663aed03","Type":"ContainerDied","Data":"914cc097afc37e9806591b01ea51f84437cc2e6d7aae5664f23d72bbe37f3f32"} Sep 07 10:22:06.375948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.375674 2577 scope.go:117] "RemoveContainer" containerID="50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6" Sep 07 10:22:06.384877 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.384857 2577 scope.go:117] "RemoveContainer" containerID="50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6" Sep 07 10:22:06.385186 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:06.385162 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6\": container with ID starting with 50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6 not found: ID does not exist" containerID="50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6" Sep 07 10:22:06.385254 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.385200 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6"} err="failed to get container status \"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6\": rpc error: code = NotFound desc = could not find container \"50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6\": container with ID starting with 50681fac4249e130c564ca9a8a97275bf790c1db5aecfac272f1cb053553e6a6 not found: ID does not exist" Sep 07 10:22:06.395789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.395760 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:22:06.398991 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.398967 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78d796d844-fzgll"] Sep 07 10:22:06.721517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:06.721488 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="394eedcf-32dd-49e6-a8d1-a873663aed03" path="/var/lib/kubelet/pods/394eedcf-32dd-49e6-a8d1-a873663aed03/volumes" Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.119414 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.119841 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="alertmanager" containerID="cri-o://3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5" gracePeriod=120 Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.119871 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-metric" containerID="cri-o://0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc" gracePeriod=120 Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.120025 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy" containerID="cri-o://5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103" gracePeriod=120 Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.120101 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-web" containerID="cri-o://14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1" gracePeriod=120 Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.120132 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="prom-label-proxy" containerID="cri-o://e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83" gracePeriod=120 Sep 07 10:22:10.120840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.120179 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="config-reloader" containerID="cri-o://9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a" gracePeriod=120 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397854 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83" exitCode=0 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397884 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc" exitCode=0 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397891 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103" exitCode=0 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397899 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a" exitCode=0 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397907 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5" exitCode=0 Sep 07 10:22:10.397947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.397922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83"} Sep 07 10:22:10.398278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.398026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc"} Sep 07 10:22:10.398278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.398054 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103"} Sep 07 10:22:10.398278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.398079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a"} Sep 07 10:22:10.398278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:10.398101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5"} Sep 07 10:22:11.366624 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.366578 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.408113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.408071 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerID="14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1" exitCode=0 Sep 07 10:22:11.408310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.408131 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1"} Sep 07 10:22:11.408310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.408165 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.408310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.408229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fe034b-281f-4ef6-bde2-67dd927f653c","Type":"ContainerDied","Data":"c08986da7922d2e841c914c947058830d50c192cec3a8f20296fdfe787f07804"} Sep 07 10:22:11.408310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.408282 2577 scope.go:117] "RemoveContainer" containerID="e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83" Sep 07 10:22:11.417226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.417204 2577 scope.go:117] "RemoveContainer" containerID="0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc" Sep 07 10:22:11.425715 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.425695 2577 scope.go:117] "RemoveContainer" containerID="5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103" Sep 07 10:22:11.433169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.433131 2577 scope.go:117] "RemoveContainer" containerID="14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1" Sep 07 10:22:11.441758 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.441734 2577 scope.go:117] "RemoveContainer" containerID="9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a" Sep 07 10:22:11.450776 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.450749 2577 scope.go:117] "RemoveContainer" containerID="3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5" Sep 07 10:22:11.458921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.458892 2577 scope.go:117] "RemoveContainer" containerID="7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5" Sep 07 10:22:11.466714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.466690 2577 scope.go:117] "RemoveContainer" containerID="e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83" Sep 07 10:22:11.467006 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.466985 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83\": container with ID starting with e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83 not found: ID does not exist" containerID="e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83" Sep 07 10:22:11.467058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467017 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83"} err="failed to get container status \"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83\": rpc error: code = NotFound desc = could not find container \"e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83\": container with ID starting with e8b2826cf7458f2698ea82fcee009734aa78f72925885db891f10d3bf1569d83 not found: ID does not exist" Sep 07 10:22:11.467058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467038 2577 scope.go:117] "RemoveContainer" containerID="0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc" Sep 07 10:22:11.467265 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.467250 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc\": container with ID starting with 0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc not found: ID does not exist" containerID="0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc" Sep 07 10:22:11.467307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467270 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc"} err="failed to get container status \"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc\": rpc error: code = NotFound desc = could not find container \"0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc\": container with ID starting with 0643b394ef8a475c897878fb34e3e4ec87ee0ca39f925d4556eaf96cbbdf09fc not found: ID does not exist" Sep 07 10:22:11.467307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467286 2577 scope.go:117] "RemoveContainer" containerID="5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103" Sep 07 10:22:11.467563 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.467546 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103\": container with ID starting with 5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103 not found: ID does not exist" containerID="5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103" Sep 07 10:22:11.467757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467568 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103"} err="failed to get container status \"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103\": rpc error: code = NotFound desc = could not find container \"5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103\": container with ID starting with 5ef64d8c5800c3e5633cfa5a70fe7642a1066ef3a7f00f71cf27513be91fd103 not found: ID does not exist" Sep 07 10:22:11.467757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467630 2577 scope.go:117] "RemoveContainer" containerID="14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1" Sep 07 10:22:11.467876 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.467858 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1\": container with ID starting with 14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1 not found: ID does not exist" containerID="14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1" Sep 07 10:22:11.467934 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467886 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1"} err="failed to get container status \"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1\": rpc error: code = NotFound desc = could not find container \"14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1\": container with ID starting with 14f6be5264057172fbbf369634d36e02b1e6c1f53e0075f3478d19feb6ceccb1 not found: ID does not exist" Sep 07 10:22:11.467934 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.467909 2577 scope.go:117] "RemoveContainer" containerID="9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a" Sep 07 10:22:11.468204 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.468187 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a\": container with ID starting with 9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a not found: ID does not exist" containerID="9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a" Sep 07 10:22:11.468246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.468210 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a"} err="failed to get container status \"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a\": rpc error: code = NotFound desc = could not find container \"9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a\": container with ID starting with 9adc58cd0f455e0184b166eaf95438b44b7f70ea78562dbe98a2b58af7f4dc9a not found: ID does not exist" Sep 07 10:22:11.468246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.468224 2577 scope.go:117] "RemoveContainer" containerID="3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5" Sep 07 10:22:11.468430 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.468409 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5\": container with ID starting with 3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5 not found: ID does not exist" containerID="3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5" Sep 07 10:22:11.468491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.468437 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5"} err="failed to get container status \"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5\": rpc error: code = NotFound desc = could not find container \"3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5\": container with ID starting with 3d6b69cc2d8369ee2eb12e33ce94af39e41cdd2e0d98044815f6eaeaa4fc86d5 not found: ID does not exist" Sep 07 10:22:11.468491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.468456 2577 scope.go:117] "RemoveContainer" containerID="7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5" Sep 07 10:22:11.468737 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:11.468721 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5\": container with ID starting with 7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5 not found: ID does not exist" containerID="7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5" Sep 07 10:22:11.468777 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.468743 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5"} err="failed to get container status \"7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5\": rpc error: code = NotFound desc = could not find container \"7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5\": container with ID starting with 7a78e5260b410219b92685248554c4e8c11cb61580e7530125ad6d70ab780df5 not found: ID does not exist" Sep 07 10:22:11.497113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497072 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497174 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497235 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497266 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497290 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpg2x\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497368 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497394 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497445 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497476 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497501 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c0fe034b-281f-4ef6-bde2-67dd927f653c\" (UID: \"c0fe034b-281f-4ef6-bde2-67dd927f653c\") " Sep 07 10:22:11.497872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:11.497872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497706 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:11.497872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:22:11.498177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497908 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-main-db\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.498177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497928 2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.498177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.497942 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe034b-281f-4ef6-bde2-67dd927f653c-metrics-client-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.501645 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.501608 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out" (OuterVolumeSpecName: "config-out") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:22:11.501802 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.501609 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.502114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502067 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:22:11.502214 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502115 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.502214 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502178 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x" (OuterVolumeSpecName: "kube-api-access-wpg2x") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "kube-api-access-wpg2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:22:11.502214 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502199 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.502341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502321 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.502602 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.502556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume" (OuterVolumeSpecName: "config-volume") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.505862 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.505831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.512648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.512619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config" (OuterVolumeSpecName: "web-config") pod "c0fe034b-281f-4ef6-bde2-67dd927f653c" (UID: "c0fe034b-281f-4ef6-bde2-67dd927f653c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:11.599133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599090 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-main-tls\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599151 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-tls-assets\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599161 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpg2x\" (UniqueName: \"kubernetes.io/projected/c0fe034b-281f-4ef6-bde2-67dd927f653c-kube-api-access-wpg2x\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599170 2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-cluster-tls-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599179 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599189 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599199 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-web-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599208 2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599215 2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fe034b-281f-4ef6-bde2-67dd927f653c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.599305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.599224 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fe034b-281f-4ef6-bde2-67dd927f653c-config-out\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:11.731292 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.731254 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:11.736929 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.736899 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:11.762728 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.762689 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:11.763014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763002 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="config-reloader" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763016 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="config-reloader" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763025 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-metric" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763030 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-metric" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763040 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08228660-eb6b-4682-a4da-c9ad862f71db" containerName="console" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763046 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08228660-eb6b-4682-a4da-c9ad862f71db" containerName="console" Sep 07 10:22:11.763054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763054 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="init-config-reloader" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763060 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="init-config-reloader" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763072 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="394eedcf-32dd-49e6-a8d1-a873663aed03" containerName="console" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763078 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="394eedcf-32dd-49e6-a8d1-a873663aed03" containerName="console" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763085 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="alertmanager" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763090 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="alertmanager" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763103 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-web" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763108 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-web" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763113 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763118 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763124 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="prom-label-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763128 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="prom-label-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763177 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="alertmanager" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763186 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763191 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="prom-label-proxy" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763197 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="config-reloader" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763202 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08228660-eb6b-4682-a4da-c9ad862f71db" containerName="console" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763210 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-web" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763217 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="394eedcf-32dd-49e6-a8d1-a873663aed03" containerName="console" Sep 07 10:22:11.763222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.763223 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" containerName="kube-rbac-proxy-metric" Sep 07 10:22:11.767320 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.767286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.769557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.769524 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 07 10:22:11.769736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.769611 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 07 10:22:11.769853 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.769739 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 07 10:22:11.769853 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.769525 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 07 10:22:11.770100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.770081 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 07 10:22:11.770192 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.770157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hjpgr\"" Sep 07 10:22:11.770253 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.770221 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 07 10:22:11.770383 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.770358 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 07 10:22:11.770433 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.770407 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 07 10:22:11.775071 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.775050 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 07 10:22:11.779504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.779479 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:11.901852 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.901810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-config-out\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.901865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.901903 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.901977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh86t\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-kube-api-access-zh86t\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-web-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902265 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:11.902402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:11.902301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.002942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.002904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-web-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.002946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.002976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-config-out\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh86t\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-kube-api-access-zh86t\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.003998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.003565 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.004174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.004148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.004907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.004878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5366bba2-fa89-49f6-b869-08ef034d58ff-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.006432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.006388 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5366bba2-fa89-49f6-b869-08ef034d58ff-config-out\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.006854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.006710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.006854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.006734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.006854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.006802 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.006854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.006848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-web-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.007265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.007245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.007644 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.007624 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.007705 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.007692 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.008088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.008071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5366bba2-fa89-49f6-b869-08ef034d58ff-config-volume\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.011032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.011011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh86t\" (UniqueName: \"kubernetes.io/projected/5366bba2-fa89-49f6-b869-08ef034d58ff-kube-api-access-zh86t\") pod \"alertmanager-main-0\" (UID: \"5366bba2-fa89-49f6-b869-08ef034d58ff\") " pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.078809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.078771 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 07 10:22:12.358159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.358071 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 07 10:22:12.361215 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:22:12.361188 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5366bba2_fa89_49f6_b869_08ef034d58ff.slice/crio-a5873739428fd06774b7bf2991944d9cb5b0fc837b7f052678c41205f27c0540 WatchSource:0}: Error finding container a5873739428fd06774b7bf2991944d9cb5b0fc837b7f052678c41205f27c0540: Status 404 returned error can't find the container with id a5873739428fd06774b7bf2991944d9cb5b0fc837b7f052678c41205f27c0540 Sep 07 10:22:12.415935 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.415894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"a5873739428fd06774b7bf2991944d9cb5b0fc837b7f052678c41205f27c0540"} Sep 07 10:22:12.722607 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:12.722560 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0fe034b-281f-4ef6-bde2-67dd927f653c" path="/var/lib/kubelet/pods/c0fe034b-281f-4ef6-bde2-67dd927f653c/volumes" Sep 07 10:22:13.423863 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:13.423830 2577 generic.go:356] "Generic (PLEG): container finished" podID="5366bba2-fa89-49f6-b869-08ef034d58ff" containerID="209a835feb9ae5e94d77cbcd6017c046814f2dbfc299820f22a5e071e1f5b38b" exitCode=0 Sep 07 10:22:13.424262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:13.423884 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerDied","Data":"209a835feb9ae5e94d77cbcd6017c046814f2dbfc299820f22a5e071e1f5b38b"} Sep 07 10:22:14.360506 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.360471 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:14.360873 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.360824 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="prometheus" containerID="cri-o://cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" gracePeriod=600 Sep 07 10:22:14.360965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.360903 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" gracePeriod=600 Sep 07 10:22:14.361054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.361032 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-web" containerID="cri-o://0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" gracePeriod=600 Sep 07 10:22:14.361114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.361069 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="config-reloader" containerID="cri-o://af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" gracePeriod=600 Sep 07 10:22:14.361114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.360842 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy" containerID="cri-o://b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" gracePeriod=600 Sep 07 10:22:14.361203 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.361160 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="thanos-sidecar" containerID="cri-o://d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" gracePeriod=600 Sep 07 10:22:14.437342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"f8dea4c5bccc183b54bbbc8841f4c486b72e97ce458848068542fce2266a156c"} Sep 07 10:22:14.437739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"aa155509872cfd05779f75d32ebfc2c11432f15bb18c282fd939f6ba58468f4e"} Sep 07 10:22:14.437739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437430 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"68c2d6b5bc3307b63d176a2048698272ad65126a173491e7d0c69e6eabaa22fd"} Sep 07 10:22:14.437739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"91af2eca5b6bd16f8142ac35f518bedf1397edd09b55c39018dd1ad86b10ea7e"} Sep 07 10:22:14.437739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"0d70dc3efb872f2adbc3cc631015a2013a1346209b9d133b958fa8b70f5f966b"} Sep 07 10:22:14.437739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.437502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5366bba2-fa89-49f6-b869-08ef034d58ff","Type":"ContainerStarted","Data":"400d505c0dc3af8f15e94ad980a9ab7f7c5653088d1e99ab1e080a2e979adaed"} Sep 07 10:22:14.461270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.461212 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.447763168 podStartE2EDuration="3.461194274s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.431106ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:14.45965645 +0000 UTC m=+102.317345767" watchObservedRunningTime="2026-09-07 10:22:14.461194274 +0000 UTC m=+102.318883660" Sep 07 10:22:14.618542 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.618484 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:14.728304 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728273 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728304 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728306 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728516 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728337 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728516 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728372 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728617 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728651 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728734 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728759 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728790 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.728816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728814 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728840 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728878 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728921 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l8kz\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728956 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.728981 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.729012 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.729053 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.729099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config\") pod \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\" (UID: \"2b0e7d70-eea4-4dca-9b87-3fb05663c60c\") " Sep 07 10:22:14.729898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.729573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:22:14.730121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.729894 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.730121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.730026 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-db\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.730121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.730045 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.730297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.730273 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.731246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.730927 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.731706 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.731639 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.731706 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.731706 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.732000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.731958 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.732342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.732315 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:22:14.732342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.732330 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out" (OuterVolumeSpecName: "config-out") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:22:14.732470 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.732441 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config" (OuterVolumeSpecName: "config") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.732661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.732632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.732865 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.732837 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.733184 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.733154 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.733312 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.733292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.733819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.733783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.733969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.733942 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.734709 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.734685 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:22:14.734793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.734722 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.734793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.734777 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz" (OuterVolumeSpecName: "kube-api-access-8l8kz") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "kube-api-access-8l8kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:22:14.745385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.745349 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config" (OuterVolumeSpecName: "web-config") pod "2b0e7d70-eea4-4dca-9b87-3fb05663c60c" (UID: "2b0e7d70-eea4-4dca-9b87-3fb05663c60c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:22:14.831413 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831371 2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831413 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831407 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831413 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831420 2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-kube-rbac-proxy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831434 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831446 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831458 2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-tls-assets\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831468 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831479 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8kz\" (UniqueName: \"kubernetes.io/projected/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-kube-api-access-8l8kz\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831490 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831503 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831517 2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-configmap-metrics-client-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831529 2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831541 2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-web-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831553 2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831566 2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config-out\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831578 2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-config\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831621 2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-grpc-tls\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:14.831718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:14.831632 2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2b0e7d70-eea4-4dca-9b87-3fb05663c60c-secret-metrics-client-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:22:15.281519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.281486 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-97grq" Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445145 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" exitCode=0 Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445174 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" exitCode=0 Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445187 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" exitCode=0 Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445194 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" exitCode=0 Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445201 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" exitCode=0 Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} Sep 07 10:22:15.445239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445219 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445208 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" exitCode=0 Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445320 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445354 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445357 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} Sep 07 10:22:15.445969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.445551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2b0e7d70-eea4-4dca-9b87-3fb05663c60c","Type":"ContainerDied","Data":"a95b0326bc74043a1516c774f4cfdbda1c29f9c6ce077b410f7c0303efc55982"} Sep 07 10:22:15.453767 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.453752 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.460935 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.460918 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.467511 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.467487 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:15.469133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.469114 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.472387 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.472367 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:15.477154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.477011 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.484254 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.484235 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.492401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.492381 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.500701 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.500660 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:15.500781 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.500762 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.501057 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501041 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="init-config-reloader" Sep 07 10:22:15.501096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501059 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="init-config-reloader" Sep 07 10:22:15.501096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501071 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-thanos" Sep 07 10:22:15.501096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501076 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-thanos" Sep 07 10:22:15.501096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="prometheus" Sep 07 10:22:15.501096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501092 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="prometheus" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501098 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-web" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.501096 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501131 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501150 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501104 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-web" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501187 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="thanos-sidecar" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501192 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="thanos-sidecar" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501199 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="config-reloader" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501204 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="config-reloader" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501210 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy" Sep 07 10:22:15.501255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501215 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501292 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="config-reloader" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-thanos" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501321 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy-web" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501327 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="prometheus" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501332 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="thanos-sidecar" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" containerName="kube-rbac-proxy" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.501461 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501483 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.501654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501500 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.501925 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.501799 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.501925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501819 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.501925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.501834 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.502078 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.502055 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.502110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502087 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.502142 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502107 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.502369 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.502351 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.502421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502371 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.502421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502384 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.502665 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.502648 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.502705 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502669 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.502705 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502683 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.502932 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:22:15.502916 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.502976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502933 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.502976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.502948 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.503173 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503157 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.503215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503173 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.503389 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503371 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.503432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503390 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.503634 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503611 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.503634 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503633 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.503849 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503830 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.503959 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.503850 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.504156 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504132 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.504206 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504160 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.504424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504394 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.504424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504422 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.504809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504788 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.504809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.504807 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.505073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505053 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.505121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505075 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.505301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505283 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.505345 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505301 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.505562 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505541 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.505644 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505562 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.505855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505833 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.505855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.505855 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.506151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506130 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.506209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506152 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.506375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506357 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.506448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506377 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.506626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506608 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.506678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506627 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.506841 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506826 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.506886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.506844 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.507114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507098 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.507159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507114 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.507359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507342 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.507399 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507361 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.507639 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507621 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.507690 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507640 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.507846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507827 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.507889 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.507847 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.508078 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508057 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.508119 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508079 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.508344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508327 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.508387 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508345 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.508554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508534 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.508639 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508556 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.508803 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508787 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.508861 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508803 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.509017 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.508999 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.509081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509018 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.509213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509198 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.509250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509213 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.509449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509429 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.509488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509460 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.509730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509710 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.509777 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509731 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.509991 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509975 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.509991 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.509991 2577 scope.go:117] "RemoveContainer" containerID="47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f" Sep 07 10:22:15.510220 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510202 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f"} err="failed to get container status \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": rpc error: code = NotFound desc = could not find container \"47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f\": container with ID starting with 47bee41f537295ebfc4b9b72e8c8d835438f736ea8186c214aaa5e14b84a289f not found: ID does not exist" Sep 07 10:22:15.510286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510220 2577 scope.go:117] "RemoveContainer" containerID="b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588" Sep 07 10:22:15.510486 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510426 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588"} err="failed to get container status \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": rpc error: code = NotFound desc = could not find container \"b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588\": container with ID starting with b6f3ace92ce429269a03b8ab083528f82079b888e4b599ad1eedeb02c6c08588 not found: ID does not exist" Sep 07 10:22:15.510486 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510485 2577 scope.go:117] "RemoveContainer" containerID="0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9" Sep 07 10:22:15.510837 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510819 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9"} err="failed to get container status \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": rpc error: code = NotFound desc = could not find container \"0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9\": container with ID starting with 0d240252ccc23e8405168537831779429d94a68e4a0976586825e004285ac3e9 not found: ID does not exist" Sep 07 10:22:15.510891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.510838 2577 scope.go:117] "RemoveContainer" containerID="d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756" Sep 07 10:22:15.511124 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511104 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756"} err="failed to get container status \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": rpc error: code = NotFound desc = could not find container \"d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756\": container with ID starting with d2fae6da2ada32563506a0ddf1f7e9664d0ca1e0ff973a5c8d7ae6f91e212756 not found: ID does not exist" Sep 07 10:22:15.511213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511125 2577 scope.go:117] "RemoveContainer" containerID="af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184" Sep 07 10:22:15.511383 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511362 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184"} err="failed to get container status \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": rpc error: code = NotFound desc = could not find container \"af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184\": container with ID starting with af502985b6c2d2b8f8f8ba508c98f9843089eb2e093992d10a0561df1eb48184 not found: ID does not exist" Sep 07 10:22:15.511429 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511384 2577 scope.go:117] "RemoveContainer" containerID="cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8" Sep 07 10:22:15.511648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511630 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8"} err="failed to get container status \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": rpc error: code = NotFound desc = could not find container \"cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8\": container with ID starting with cd3d9433429291f1a46ee6dbf44854e0f97845531ad4ebb5cbd9f35bd70d97b8 not found: ID does not exist" Sep 07 10:22:15.511711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511649 2577 scope.go:117] "RemoveContainer" containerID="68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c" Sep 07 10:22:15.511893 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.511872 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c"} err="failed to get container status \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": rpc error: code = NotFound desc = could not find container \"68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c\": container with ID starting with 68d590dec8cc68689a8dabd139cfc19266908f39c1059175372d23181362c38c not found: ID does not exist" Sep 07 10:22:15.534336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.534252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:15.534702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.534433 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.537987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.537891 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 07 10:22:15.538273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.538025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.538850 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.538918 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.538999 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dlam26pi8q80t\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539097 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539119 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539200 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539257 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 07 10:22:15.539401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539279 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 07 10:22:15.540188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539613 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 07 10:22:15.540188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539629 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 07 10:22:15.540188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.539820 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 07 10:22:15.540965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.540921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6mgl\"" Sep 07 10:22:15.552734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.546379 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 07 10:22:15.552734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.551857 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 07 10:22:15.640507 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640507 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640678 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.640908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drjr6\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-kube-api-access-drjr6\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640917 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.640999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.641013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.641037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.641063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.641217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.641090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742289 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742352 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742435 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.742772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drjr6\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-kube-api-access-drjr6\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.742999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.743020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.743391 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.743241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.745063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.744011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.745063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.744736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746029 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.745993 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746029 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-web-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746048 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746611 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746611 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.746906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.746636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.747365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.747063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.748490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.748458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config-out\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.748657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.748639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.749467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.749440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.749543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.749445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bd89dd7b-df6a-4d46-a130-45c53d304fdd-config\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.749738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.749721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bd89dd7b-df6a-4d46-a130-45c53d304fdd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.754346 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.754324 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drjr6\" (UniqueName: \"kubernetes.io/projected/bd89dd7b-df6a-4d46-a130-45c53d304fdd-kube-api-access-drjr6\") pod \"prometheus-k8s-0\" (UID: \"bd89dd7b-df6a-4d46-a130-45c53d304fdd\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:15.858073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:15.858030 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:22:16.160257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:16.160228 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 07 10:22:16.162727 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:22:16.162693 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd89dd7b_df6a_4d46_a130_45c53d304fdd.slice/crio-c4c7fc2a48bd7b2a2e62399e2cbdc6eec3ea6852d1c805c105c336f28232b388 WatchSource:0}: Error finding container c4c7fc2a48bd7b2a2e62399e2cbdc6eec3ea6852d1c805c105c336f28232b388: Status 404 returned error can't find the container with id c4c7fc2a48bd7b2a2e62399e2cbdc6eec3ea6852d1c805c105c336f28232b388 Sep 07 10:22:16.453342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:16.453253 2577 generic.go:356] "Generic (PLEG): container finished" podID="bd89dd7b-df6a-4d46-a130-45c53d304fdd" containerID="6e07b12316ad8bcea62383fe0bb1e0a2758901800e7912326fc1c007fd228138" exitCode=0 Sep 07 10:22:16.453342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:16.453328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerDied","Data":"6e07b12316ad8bcea62383fe0bb1e0a2758901800e7912326fc1c007fd228138"} Sep 07 10:22:16.453759 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:16.453389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"c4c7fc2a48bd7b2a2e62399e2cbdc6eec3ea6852d1c805c105c336f28232b388"} Sep 07 10:22:16.722817 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:16.722749 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b0e7d70-eea4-4dca-9b87-3fb05663c60c" path="/var/lib/kubelet/pods/2b0e7d70-eea4-4dca-9b87-3fb05663c60c/volumes" Sep 07 10:22:17.464209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"934e2158c8e682d1e29e6815dc1cfcb242156807f33e26fbe5d7a7f4f68d413f"} Sep 07 10:22:17.464209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"a55705dc89bdf642e7b20fd0f3b5347b54cf3b91d63840cebe129c4fb99b0989"} Sep 07 10:22:17.464669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"9e920eb7b3d1a0b35b3480cd633c3ad404fd09f4eb58104ea75841e4d2535def"} Sep 07 10:22:17.464669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"2701ce0edb1edf22379ebecdb67a0864b99975aec116427ba41eb5a4535b814d"} Sep 07 10:22:17.464669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"3b3df05e9fcb7a54914f3602abd646daa955156719ad2baf8f28784bc9dbb8c2"} Sep 07 10:22:17.464669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.464297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bd89dd7b-df6a-4d46-a130-45c53d304fdd","Type":"ContainerStarted","Data":"94aa74e13bb3a9e4bf33f0e8a2d3bcd2728956edf6c7215b00b03baa5c73eb53"} Sep 07 10:22:17.501272 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:17.501218 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.478875263 podStartE2EDuration="2.501200819s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.325556ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:17.499273691 +0000 UTC m=+105.356963000" watchObservedRunningTime="2026-09-07 10:22:17.501200819 +0000 UTC m=+105.358890136" Sep 07 10:22:20.859029 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:22:20.858979 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:23:15.858425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:23:15.858378 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:23:15.875558 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:23:15.875526 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:23:16.721706 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:23:16.721682 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 07 10:25:32.599003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:25:32.598973 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 07 10:31:01.236143 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.236055 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g4gkk"] Sep 07 10:31:01.238549 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.238527 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.240631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.240607 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 07 10:31:01.240785 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.240769 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 07 10:31:01.241224 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.241208 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-r679s\"" Sep 07 10:31:01.248710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.248686 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g4gkk"] Sep 07 10:31:01.404085 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.404045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.404265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.404091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds26f\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-kube-api-access-ds26f\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.505483 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.505401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.505483 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.505443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds26f\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-kube-api-access-ds26f\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.513663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.513630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.513791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.513685 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds26f\" (UniqueName: \"kubernetes.io/projected/0af2271f-7d1a-40bf-8b89-19024e7b61e6-kube-api-access-ds26f\") pod \"cert-manager-webhook-6486ffd997-g4gkk\" (UID: \"0af2271f-7d1a-40bf-8b89-19024e7b61e6\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.568369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.568332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:01.842841 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.842761 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g4gkk"] Sep 07 10:31:01.846046 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:31:01.846009 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0af2271f_7d1a_40bf_8b89_19024e7b61e6.slice/crio-2213398822d897f1cb2d903733fb6c805c94972e50e7188a692bfca5890e4b37 WatchSource:0}: Error finding container 2213398822d897f1cb2d903733fb6c805c94972e50e7188a692bfca5890e4b37: Status 404 returned error can't find the container with id 2213398822d897f1cb2d903733fb6c805c94972e50e7188a692bfca5890e4b37 Sep 07 10:31:01.848323 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:01.848304 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:31:02.563904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:02.563856 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" event={"ID":"0af2271f-7d1a-40bf-8b89-19024e7b61e6","Type":"ContainerStarted","Data":"2213398822d897f1cb2d903733fb6c805c94972e50e7188a692bfca5890e4b37"} Sep 07 10:31:05.579976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:05.579941 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" event={"ID":"0af2271f-7d1a-40bf-8b89-19024e7b61e6","Type":"ContainerStarted","Data":"dfb152283ca76bd3526fc2fb920505497f9ffca4b177bdf547577bb19f2448ad"} Sep 07 10:31:05.580420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:05.580063 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:05.597643 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:05.597563 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" podStartSLOduration=1.56922497 podStartE2EDuration="4.597544557s" totalImagesPullingTime="3.028319587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:05.595883563 +0000 UTC m=+633.453572878" watchObservedRunningTime="2026-09-07 10:31:05.597544557 +0000 UTC m=+633.455233873" Sep 07 10:31:11.588402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:11.588369 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g4gkk" Sep 07 10:31:13.658912 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.658881 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68"] Sep 07 10:31:13.661117 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.661096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.663058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.663031 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 07 10:31:13.663627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.663562 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 07 10:31:13.663741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.663565 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7b4xq\"" Sep 07 10:31:13.669761 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.669738 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68"] Sep 07 10:31:13.711503 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.711469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10431c8d-ba99-4946-b33a-1f238c7d8362-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.711717 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.711530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdsd\" (UniqueName: \"kubernetes.io/projected/10431c8d-ba99-4946-b33a-1f238c7d8362-kube-api-access-fkdsd\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.812961 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.812919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10431c8d-ba99-4946-b33a-1f238c7d8362-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.813142 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.812986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdsd\" (UniqueName: \"kubernetes.io/projected/10431c8d-ba99-4946-b33a-1f238c7d8362-kube-api-access-fkdsd\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.813357 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.813333 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/10431c8d-ba99-4946-b33a-1f238c7d8362-tmp\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.821739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.821707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdsd\" (UniqueName: \"kubernetes.io/projected/10431c8d-ba99-4946-b33a-1f238c7d8362-kube-api-access-fkdsd\") pod \"openshift-lws-operator-fd8ccff4c-x4w68\" (UID: \"10431c8d-ba99-4946-b33a-1f238c7d8362\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:13.971798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:13.971759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" Sep 07 10:31:14.244657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:14.244560 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68"] Sep 07 10:31:14.248167 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:31:14.248142 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10431c8d_ba99_4946_b33a_1f238c7d8362.slice/crio-91b434a7c7efea8c1bc6f8099848fb25e5e9ab3025b7f93bc7752f0a0f96389d WatchSource:0}: Error finding container 91b434a7c7efea8c1bc6f8099848fb25e5e9ab3025b7f93bc7752f0a0f96389d: Status 404 returned error can't find the container with id 91b434a7c7efea8c1bc6f8099848fb25e5e9ab3025b7f93bc7752f0a0f96389d Sep 07 10:31:14.622077 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:14.621987 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" event={"ID":"10431c8d-ba99-4946-b33a-1f238c7d8362","Type":"ContainerStarted","Data":"91b434a7c7efea8c1bc6f8099848fb25e5e9ab3025b7f93bc7752f0a0f96389d"} Sep 07 10:31:17.640648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:17.640609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" event={"ID":"10431c8d-ba99-4946-b33a-1f238c7d8362","Type":"ContainerStarted","Data":"bda1ef99cb4508dc6c8f732ef9e52c2ebe627f1b741f9e26a11bc147dccbc353"} Sep 07 10:31:17.656536 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:17.656481 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-x4w68" podStartSLOduration=2.201621098 podStartE2EDuration="4.656462519s" totalImagesPullingTime="2.454841421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:17.655728076 +0000 UTC m=+645.513417394" watchObservedRunningTime="2026-09-07 10:31:17.656462519 +0000 UTC m=+645.514151834" Sep 07 10:31:36.160388 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.160346 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc"] Sep 07 10:31:36.166171 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.166150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.171830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.171808 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 07 10:31:36.174100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.174075 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 07 10:31:36.174661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.174641 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t7dj9\"" Sep 07 10:31:36.174773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.174643 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 07 10:31:36.174773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.174720 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 07 10:31:36.205235 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.205200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc"] Sep 07 10:31:36.323232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.323194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.323452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.323245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qhn\" (UniqueName: \"kubernetes.io/projected/87d3670a-51d7-4798-ae27-eeddba6b32d2-kube-api-access-z5qhn\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.323452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.323343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.323452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.323380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.323452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.323439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424272 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424272 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qhn\" (UniqueName: \"kubernetes.io/projected/87d3670a-51d7-4798-ae27-eeddba6b32d2-kube-api-access-z5qhn\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.424856 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.424829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-manifests\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.426725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.426702 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d3670a-51d7-4798-ae27-eeddba6b32d2-tmp\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.426948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.426930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.426994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.426979 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/87d3670a-51d7-4798-ae27-eeddba6b32d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.432442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.432420 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qhn\" (UniqueName: \"kubernetes.io/projected/87d3670a-51d7-4798-ae27-eeddba6b32d2-kube-api-access-z5qhn\") pod \"opendatahub-operator-controller-manager-7f5bfcc886-jc5vc\" (UID: \"87d3670a-51d7-4798-ae27-eeddba6b32d2\") " pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.476469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.476434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:36.754958 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:36.754907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc"] Sep 07 10:31:36.759091 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:31:36.759061 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d3670a_51d7_4798_ae27_eeddba6b32d2.slice/crio-1bec0d877994de656e1520e552613c441052681aec6055f4fc05b9052997a601 WatchSource:0}: Error finding container 1bec0d877994de656e1520e552613c441052681aec6055f4fc05b9052997a601: Status 404 returned error can't find the container with id 1bec0d877994de656e1520e552613c441052681aec6055f4fc05b9052997a601 Sep 07 10:31:37.728790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:37.728742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" event={"ID":"87d3670a-51d7-4798-ae27-eeddba6b32d2","Type":"ContainerStarted","Data":"1bec0d877994de656e1520e552613c441052681aec6055f4fc05b9052997a601"} Sep 07 10:31:40.744420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:40.744382 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" event={"ID":"87d3670a-51d7-4798-ae27-eeddba6b32d2","Type":"ContainerStarted","Data":"42dddaeeaaf8b55234e402c0473e4e8c60ea1bf6ccaf4e12f0570bb0176b4126"} Sep 07 10:31:41.750040 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:41.749996 2577 generic.go:356] "Generic (PLEG): container finished" podID="87d3670a-51d7-4798-ae27-eeddba6b32d2" containerID="42dddaeeaaf8b55234e402c0473e4e8c60ea1bf6ccaf4e12f0570bb0176b4126" exitCode=0 Sep 07 10:31:41.750641 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:41.750049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" event={"ID":"87d3670a-51d7-4798-ae27-eeddba6b32d2","Type":"ContainerDied","Data":"42dddaeeaaf8b55234e402c0473e4e8c60ea1bf6ccaf4e12f0570bb0176b4126"} Sep 07 10:31:42.756257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:42.756219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" event={"ID":"87d3670a-51d7-4798-ae27-eeddba6b32d2","Type":"ContainerStarted","Data":"6f8c56e1aa2905ea0e1cd6b391a4346a0bf9342a6e90773dda910915a3f27770"} Sep 07 10:31:42.756664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:42.756383 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:31:42.780255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:42.780197 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" podStartSLOduration=2.447840974 podStartE2EDuration="6.780179136s" totalImagesPullingTime="3.269473377s" totalInitContainerRuntime="1.062864785s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:42.777976955 +0000 UTC m=+670.635666283" watchObservedRunningTime="2026-09-07 10:31:42.780179136 +0000 UTC m=+670.637868451" Sep 07 10:31:53.763368 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:31:53.763329 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f5bfcc886-jc5vc" Sep 07 10:32:01.050550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.050515 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn"] Sep 07 10:32:01.053762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.053738 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.056480 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.056451 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 07 10:32:01.056662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.056453 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 07 10:32:01.056662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.056453 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 07 10:32:01.056662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.056500 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qgd92\"" Sep 07 10:32:01.062271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.062245 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn"] Sep 07 10:32:01.140995 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.140956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.141185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.141004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3983afd-47b6-4621-994a-f69c7784cfd4-manager-config\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.141185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.141092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-metrics-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.141185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.141142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnxlr\" (UniqueName: \"kubernetes.io/projected/b3983afd-47b6-4621-994a-f69c7784cfd4-kube-api-access-xnxlr\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.242432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.242389 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3983afd-47b6-4621-994a-f69c7784cfd4-manager-config\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.242660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.242447 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-metrics-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.242660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.242564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnxlr\" (UniqueName: \"kubernetes.io/projected/b3983afd-47b6-4621-994a-f69c7784cfd4-kube-api-access-xnxlr\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.242771 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.242691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.243308 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.243285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3983afd-47b6-4621-994a-f69c7784cfd4-manager-config\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.245266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.245231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.245348 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.245297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3983afd-47b6-4621-994a-f69c7784cfd4-metrics-cert\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.250477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.250439 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnxlr\" (UniqueName: \"kubernetes.io/projected/b3983afd-47b6-4621-994a-f69c7784cfd4-kube-api-access-xnxlr\") pod \"lws-controller-manager-547fb475d5-8lrwn\" (UID: \"b3983afd-47b6-4621-994a-f69c7784cfd4\") " pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.364757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.364661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:01.643344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.643265 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn"] Sep 07 10:32:01.645631 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:32:01.645596 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3983afd_47b6_4621_994a_f69c7784cfd4.slice/crio-a8a16346deb106615ab41baf3df5f1be2ed0c891e6197bc9b464a0e0d656cf89 WatchSource:0}: Error finding container a8a16346deb106615ab41baf3df5f1be2ed0c891e6197bc9b464a0e0d656cf89: Status 404 returned error can't find the container with id a8a16346deb106615ab41baf3df5f1be2ed0c891e6197bc9b464a0e0d656cf89 Sep 07 10:32:01.843497 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:01.843457 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" event={"ID":"b3983afd-47b6-4621-994a-f69c7784cfd4","Type":"ContainerStarted","Data":"a8a16346deb106615ab41baf3df5f1be2ed0c891e6197bc9b464a0e0d656cf89"} Sep 07 10:32:03.053209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.053177 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49"] Sep 07 10:32:03.057514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.057494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.059725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.059697 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 07 10:32:03.059725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.059710 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5pddb\"" Sep 07 10:32:03.059907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.059699 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 07 10:32:03.059907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.059700 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 07 10:32:03.069305 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.069275 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49"] Sep 07 10:32:03.158739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158739 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnxs5\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-kube-api-access-pnxs5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.158950 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158933 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.159177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.158970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnxs5\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-kube-api-access-pnxs5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.259808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.259919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.260038 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260236 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.260086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260236 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.260226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260349 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.260306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.260385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.260372 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.262202 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.262181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.262276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.262262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.268565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.268541 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnxs5\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-kube-api-access-pnxs5\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.268701 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.268567 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe2c7b7-4459-4f6a-aaac-a672306eefd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49\" (UID: \"ffe2c7b7-4459-4f6a-aaac-a672306eefd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.369918 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.369825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:03.647721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.647597 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49"] Sep 07 10:32:03.650551 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:32:03.650520 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe2c7b7_4459_4f6a_aaac_a672306eefd3.slice/crio-14c9574a5ea977918739d1618dd7f7ebb199b4d7f26ba25d4a0abca70793ef26 WatchSource:0}: Error finding container 14c9574a5ea977918739d1618dd7f7ebb199b4d7f26ba25d4a0abca70793ef26: Status 404 returned error can't find the container with id 14c9574a5ea977918739d1618dd7f7ebb199b4d7f26ba25d4a0abca70793ef26 Sep 07 10:32:03.854578 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:03.854535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" event={"ID":"ffe2c7b7-4459-4f6a-aaac-a672306eefd3","Type":"ContainerStarted","Data":"14c9574a5ea977918739d1618dd7f7ebb199b4d7f26ba25d4a0abca70793ef26"} Sep 07 10:32:06.498895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.498859 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:32:06.499155 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.498953 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:32:06.499155 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.498979 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:32:06.563891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.563865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xph9l"] Sep 07 10:32:06.567216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.567194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.569301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.569255 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 07 10:32:06.569716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.569376 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t99sr\"" Sep 07 10:32:06.577139 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.577059 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xph9l"] Sep 07 10:32:06.693350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.693309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.693549 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.693405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mk9h\" (UniqueName: \"kubernetes.io/projected/577b010a-6084-440b-b873-62af7d015b08-kube-api-access-6mk9h\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.794730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.794645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.794730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.794697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mk9h\" (UniqueName: \"kubernetes.io/projected/577b010a-6084-440b-b873-62af7d015b08-kube-api-access-6mk9h\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.794935 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:32:06.794805 2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 07 10:32:06.794935 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:32:06.794870 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert podName:577b010a-6084-440b-b873-62af7d015b08 nodeName:}" failed. No retries permitted until 2026-09-07 10:32:07.294852753 +0000 UTC m=+695.152542051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert") pod "odh-model-controller-7998c67dc4-xph9l" (UID: "577b010a-6084-440b-b873-62af7d015b08") : secret "odh-model-controller-webhook-cert" not found Sep 07 10:32:06.806041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.806011 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mk9h\" (UniqueName: \"kubernetes.io/projected/577b010a-6084-440b-b873-62af7d015b08-kube-api-access-6mk9h\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:06.871152 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.871116 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" event={"ID":"ffe2c7b7-4459-4f6a-aaac-a672306eefd3","Type":"ContainerStarted","Data":"8e1b848ce893ebaabcbac0850ba8b8af3b7dd1c7aff7f249cd686f18ad693369"} Sep 07 10:32:06.893700 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:06.893647 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" podStartSLOduration=1.0481439 podStartE2EDuration="3.893626357s" totalImagesPullingTime="2.845482457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:06.892630139 +0000 UTC m=+694.750319458" watchObservedRunningTime="2026-09-07 10:32:06.893626357 +0000 UTC m=+694.751315679" Sep 07 10:32:07.088740 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.088641 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fvdcj"] Sep 07 10:32:07.092223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.092200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.094185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.094160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 07 10:32:07.094313 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.094205 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s6l99\"" Sep 07 10:32:07.101984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.101959 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fvdcj"] Sep 07 10:32:07.198549 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.198514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p7n\" (UniqueName: \"kubernetes.io/projected/8d987cce-39d5-4cac-ba92-4fba3d49fb71-kube-api-access-w4p7n\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.198752 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.198566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d987cce-39d5-4cac-ba92-4fba3d49fb71-tls-certs\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.299652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.299610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:07.299834 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.299715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p7n\" (UniqueName: \"kubernetes.io/projected/8d987cce-39d5-4cac-ba92-4fba3d49fb71-kube-api-access-w4p7n\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.299834 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.299775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d987cce-39d5-4cac-ba92-4fba3d49fb71-tls-certs\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.302393 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.302368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d987cce-39d5-4cac-ba92-4fba3d49fb71-tls-certs\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.302498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.302425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/577b010a-6084-440b-b873-62af7d015b08-cert\") pod \"odh-model-controller-7998c67dc4-xph9l\" (UID: \"577b010a-6084-440b-b873-62af7d015b08\") " pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:07.309652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.309630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p7n\" (UniqueName: \"kubernetes.io/projected/8d987cce-39d5-4cac-ba92-4fba3d49fb71-kube-api-access-w4p7n\") pod \"model-serving-api-54d89bd5fc-fvdcj\" (UID: \"8d987cce-39d5-4cac-ba92-4fba3d49fb71\") " pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.370180 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.370077 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:07.375216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.375191 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:07.404077 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.404039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:07.483243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.483171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:07.678543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.678449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fvdcj"] Sep 07 10:32:07.682540 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:32:07.682501 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d987cce_39d5_4cac_ba92_4fba3d49fb71.slice/crio-65193eaa0b6c90f155676e3e5488455d540d70a13b7fccc913ed264ed50812c5 WatchSource:0}: Error finding container 65193eaa0b6c90f155676e3e5488455d540d70a13b7fccc913ed264ed50812c5: Status 404 returned error can't find the container with id 65193eaa0b6c90f155676e3e5488455d540d70a13b7fccc913ed264ed50812c5 Sep 07 10:32:07.759843 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.759805 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-xph9l"] Sep 07 10:32:07.763512 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:32:07.763482 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577b010a_6084_440b_b873_62af7d015b08.slice/crio-606984d26b32d0ad1a65dd01a12a60ddc00dd2917f3bda1ed8f4fc9fad817079 WatchSource:0}: Error finding container 606984d26b32d0ad1a65dd01a12a60ddc00dd2917f3bda1ed8f4fc9fad817079: Status 404 returned error can't find the container with id 606984d26b32d0ad1a65dd01a12a60ddc00dd2917f3bda1ed8f4fc9fad817079 Sep 07 10:32:07.876874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.876838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" event={"ID":"577b010a-6084-440b-b873-62af7d015b08","Type":"ContainerStarted","Data":"606984d26b32d0ad1a65dd01a12a60ddc00dd2917f3bda1ed8f4fc9fad817079"} Sep 07 10:32:07.878017 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.877992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" event={"ID":"8d987cce-39d5-4cac-ba92-4fba3d49fb71","Type":"ContainerStarted","Data":"65193eaa0b6c90f155676e3e5488455d540d70a13b7fccc913ed264ed50812c5"} Sep 07 10:32:07.878258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.878240 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:07.879396 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:07.879380 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49" Sep 07 10:32:12.915281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.915228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" event={"ID":"b3983afd-47b6-4621-994a-f69c7784cfd4","Type":"ContainerStarted","Data":"365be104be7e921b48585c052166bfc9d75cb645c0c5ef17541bcc15f11a5062"} Sep 07 10:32:12.915784 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.915296 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:12.916854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.916829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" event={"ID":"577b010a-6084-440b-b873-62af7d015b08","Type":"ContainerStarted","Data":"f99d154248d7e5542ddf4dd88dd78a01c2a267fd8f5c411910bf0295bb734609"} Sep 07 10:32:12.916960 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.916938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:12.918696 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.918669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" event={"ID":"8d987cce-39d5-4cac-ba92-4fba3d49fb71","Type":"ContainerStarted","Data":"5883c450f6b908803c520fa5ebc5d309083611e79571b44bd474bb19a950b0b0"} Sep 07 10:32:12.918804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.918734 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:32:12.935556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.935494 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" podStartSLOduration=1.560986181 podStartE2EDuration="11.935475533s" totalImagesPullingTime="10.374489352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:12.934300335 +0000 UTC m=+700.791989650" watchObservedRunningTime="2026-09-07 10:32:12.935475533 +0000 UTC m=+700.793164848" Sep 07 10:32:12.962145 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.962097 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" podStartSLOduration=2.703301745 podStartE2EDuration="6.962080848s" totalImagesPullingTime="4.258779103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:12.960292001 +0000 UTC m=+700.817981316" watchObservedRunningTime="2026-09-07 10:32:12.962080848 +0000 UTC m=+700.819770163" Sep 07 10:32:12.990852 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:12.990781 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" podStartSLOduration=1.66110836 podStartE2EDuration="5.990758955s" totalImagesPullingTime="4.329650595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:12.98996306 +0000 UTC m=+700.847652375" watchObservedRunningTime="2026-09-07 10:32:12.990758955 +0000 UTC m=+700.848448272" Sep 07 10:32:13.311186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.311150 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj"] Sep 07 10:32:13.353135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.353105 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj"] Sep 07 10:32:13.353303 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.353207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.355299 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.355273 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 07 10:32:13.355299 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.355297 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 07 10:32:13.355509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.355308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 07 10:32:13.355617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.355604 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2pfb6\"" Sep 07 10:32:13.464955 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.464922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.464963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.465004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.465039 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c98e54c-c885-47ad-b34d-e2ba075a7908-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.465083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.465103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7sx\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-kube-api-access-nt7sx\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.465360 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.465186 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7sx\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-kube-api-access-nt7sx\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.566965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.566735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c98e54c-c885-47ad-b34d-e2ba075a7908-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.567618 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.567514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.569261 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.569239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c98e54c-c885-47ad-b34d-e2ba075a7908-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.569480 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.569458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.569631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.569614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.569749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.569730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.579463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.579434 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7sx\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-kube-api-access-nt7sx\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.579571 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.579550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c98e54c-c885-47ad-b34d-e2ba075a7908-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-56ffj\" (UID: \"4c98e54c-c885-47ad-b34d-e2ba075a7908\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.662879 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.662849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:13.947398 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:13.947367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj"] Sep 07 10:32:13.947735 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:32:13.947703 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c98e54c_c885_47ad_b34d_e2ba075a7908.slice/crio-1eafa6584ebd5e2f957fe2a14c3a2e7088ba0e1295ae8a643c78bf637cf8485d WatchSource:0}: Error finding container 1eafa6584ebd5e2f957fe2a14c3a2e7088ba0e1295ae8a643c78bf637cf8485d: Status 404 returned error can't find the container with id 1eafa6584ebd5e2f957fe2a14c3a2e7088ba0e1295ae8a643c78bf637cf8485d Sep 07 10:32:14.930654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:14.930612 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" event={"ID":"4c98e54c-c885-47ad-b34d-e2ba075a7908","Type":"ContainerStarted","Data":"1eafa6584ebd5e2f957fe2a14c3a2e7088ba0e1295ae8a643c78bf637cf8485d"} Sep 07 10:32:16.213685 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:16.213647 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:32:16.214068 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:16.213726 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:32:16.942948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:16.942915 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" event={"ID":"4c98e54c-c885-47ad-b34d-e2ba075a7908","Type":"ContainerStarted","Data":"d92ef399f3aff64fcdbce3ebcb35e1cdc54d0ddcddd78164f463338e806664ec"} Sep 07 10:32:16.943157 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:16.943017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:16.972468 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:16.972410 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" podStartSLOduration=1.708864274 podStartE2EDuration="3.972390876s" totalImagesPullingTime="2.263526602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:32:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:32:16.966095698 +0000 UTC m=+704.823785015" watchObservedRunningTime="2026-09-07 10:32:16.972390876 +0000 UTC m=+704.830080192" Sep 07 10:32:17.949602 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:17.949558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-56ffj" Sep 07 10:32:23.926188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:23.926101 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-547fb475d5-8lrwn" Sep 07 10:32:23.926188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:23.926178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-xph9l" Sep 07 10:32:23.928400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:32:23.928374 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-fvdcj" Sep 07 10:34:22.233659 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.233625 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw"] Sep 07 10:34:22.236927 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.236906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.239027 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.239005 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4bcvb\"" Sep 07 10:34:22.249951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.249923 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw"] Sep 07 10:34:22.378129 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378085 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsvt\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-kube-api-access-ncsvt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378129 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378407 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.378512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.378487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479607 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsvt\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-kube-api-access-ncsvt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.479819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480069 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480069 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480069 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.479937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.480111 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.480122 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.480251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.480314 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.480515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.480493 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.482232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.482212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.482633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.482614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.490780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.490710 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.490941 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.490919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsvt\" (UniqueName: \"kubernetes.io/projected/16b4cfa1-20f7-42a8-83a3-213f7be5cbb6-kube-api-access-ncsvt\") pod \"maas-default-gateway-openshift-default-78bb78ff78-s2jpw\" (UID: \"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.547815 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.547776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:22.833981 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.833956 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw"] Sep 07 10:34:22.836494 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:34:22.836463 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b4cfa1_20f7_42a8_83a3_213f7be5cbb6.slice/crio-dee138fa766b43a1d313487ea32e808df4e6205aa074487ba2a2662966eaf6b0 WatchSource:0}: Error finding container dee138fa766b43a1d313487ea32e808df4e6205aa074487ba2a2662966eaf6b0: Status 404 returned error can't find the container with id dee138fa766b43a1d313487ea32e808df4e6205aa074487ba2a2662966eaf6b0 Sep 07 10:34:22.838557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.838523 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:34:22.838654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.838621 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:34:22.838654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:22.838650 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:34:23.554184 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:23.554150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" event={"ID":"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6","Type":"ContainerStarted","Data":"95366c6c93825558d807dc30f28ddf1646c37b9fef4ae17f62a5c4f77909639c"} Sep 07 10:34:23.554566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:23.554201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" event={"ID":"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6","Type":"ContainerStarted","Data":"dee138fa766b43a1d313487ea32e808df4e6205aa074487ba2a2662966eaf6b0"} Sep 07 10:34:23.574716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:23.574656 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podStartSLOduration=1.574634018 podStartE2EDuration="1.574634018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:23.574375439 +0000 UTC m=+831.432064756" watchObservedRunningTime="2026-09-07 10:34:23.574634018 +0000 UTC m=+831.432323335" Sep 07 10:34:24.548440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:24.548405 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:24.553321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:24.553294 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:24.558336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:24.558312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:34:24.559195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:34:24.559177 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:35:16.476465 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.476431 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:16.480042 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.480022 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.482077 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.482050 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 07 10:35:16.482196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.482073 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 07 10:35:16.482255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.482199 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 07 10:35:16.482332 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.482302 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-78xk4\"" Sep 07 10:35:16.494194 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.494168 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:16.596959 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.596931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.597094 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.596965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.597094 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.597035 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qbc\" (UniqueName: \"kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.660956 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.660922 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:35:16.664415 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.664394 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.674446 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.674424 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:35:16.697720 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.697694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qbc\" (UniqueName: \"kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.697872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.697770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.697872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.697790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.697989 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:35:16.697915 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 07 10:35:16.698043 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:35:16.697991 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert podName:7a7aab48-9afe-4827-8ac7-f3e8329a234f nodeName:}" failed. No retries permitted until 2026-09-07 10:35:17.197974289 +0000 UTC m=+885.055663586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert") pod "maas-controller-747c698f56-z2n7b" (UID: "7a7aab48-9afe-4827-8ac7-f3e8329a234f") : secret "maas-controller-webhook-cert" not found Sep 07 10:35:16.700242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.700217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.706037 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.706013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qbc\" (UniqueName: \"kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:16.798502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.798423 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.798502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.798460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbct6\" (UniqueName: \"kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.798502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.798489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.900372 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.900307 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.900372 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.900357 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbct6\" (UniqueName: \"kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.900673 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.900392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.900673 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:35:16.900555 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 07 10:35:16.900673 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:35:16.900648 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert podName:79e0c667-7c49-4080-a032-80f47d969d87 nodeName:}" failed. No retries permitted until 2026-09-07 10:35:17.400630742 +0000 UTC m=+885.258320036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert") pod "maas-controller-597fd75c55-psqw6" (UID: "79e0c667-7c49-4080-a032-80f47d969d87") : secret "maas-controller-webhook-cert" not found Sep 07 10:35:16.903180 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.903153 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:16.910322 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:16.910290 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbct6\" (UniqueName: \"kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:17.203370 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.203273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:17.205923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.205900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") pod \"maas-controller-747c698f56-z2n7b\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:17.405604 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.405555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:17.408401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.408373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") pod \"maas-controller-597fd75c55-psqw6\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:17.413778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.413755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:17.575972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.575917 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:17.686097 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.686068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:17.688883 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:35:17.688855 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7aab48_9afe_4827_8ac7_f3e8329a234f.slice/crio-05e225af809df1fbf9d31761872bf4fecc2ec60f41e2d8868a82861505511cca WatchSource:0}: Error finding container 05e225af809df1fbf9d31761872bf4fecc2ec60f41e2d8868a82861505511cca: Status 404 returned error can't find the container with id 05e225af809df1fbf9d31761872bf4fecc2ec60f41e2d8868a82861505511cca Sep 07 10:35:17.812402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.812366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-747c698f56-z2n7b" event={"ID":"7a7aab48-9afe-4827-8ac7-f3e8329a234f","Type":"ContainerStarted","Data":"05e225af809df1fbf9d31761872bf4fecc2ec60f41e2d8868a82861505511cca"} Sep 07 10:35:17.852561 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:17.852408 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:35:17.855013 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:35:17.854989 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e0c667_7c49_4080_a032_80f47d969d87.slice/crio-2bf1742290d5e6524c4c32de26db6ecdff31617c5d0e0b3bf9f2f513a690813f WatchSource:0}: Error finding container 2bf1742290d5e6524c4c32de26db6ecdff31617c5d0e0b3bf9f2f513a690813f: Status 404 returned error can't find the container with id 2bf1742290d5e6524c4c32de26db6ecdff31617c5d0e0b3bf9f2f513a690813f Sep 07 10:35:18.824268 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:18.824192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-psqw6" event={"ID":"79e0c667-7c49-4080-a032-80f47d969d87","Type":"ContainerStarted","Data":"2bf1742290d5e6524c4c32de26db6ecdff31617c5d0e0b3bf9f2f513a690813f"} Sep 07 10:35:20.836063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.836018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-747c698f56-z2n7b" event={"ID":"7a7aab48-9afe-4827-8ac7-f3e8329a234f","Type":"ContainerStarted","Data":"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad"} Sep 07 10:35:20.836517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.836190 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:20.837633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.837602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-psqw6" event={"ID":"79e0c667-7c49-4080-a032-80f47d969d87","Type":"ContainerStarted","Data":"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda"} Sep 07 10:35:20.837766 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.837706 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:20.888326 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.888274 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-747c698f56-z2n7b" podStartSLOduration=2.149701624 podStartE2EDuration="4.888255584s" totalImagesPullingTime="2.73855396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:20.885145103 +0000 UTC m=+888.742834421" watchObservedRunningTime="2026-09-07 10:35:20.888255584 +0000 UTC m=+888.745944899" Sep 07 10:35:20.942008 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:20.941959 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-597fd75c55-psqw6" podStartSLOduration=2.3762206470000002 podStartE2EDuration="4.941942503s" totalImagesPullingTime="2.565721856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:20.941257061 +0000 UTC m=+888.798946375" watchObservedRunningTime="2026-09-07 10:35:20.941942503 +0000 UTC m=+888.799631818" Sep 07 10:35:31.845020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:31.844937 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:31.845464 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:31.845021 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:35:31.907203 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:31.907173 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:31.907382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:31.907352 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-747c698f56-z2n7b" podUID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" containerName="manager" containerID="cri-o://b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad" gracePeriod=30 Sep 07 10:35:32.159656 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.159635 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:32.247335 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.247302 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4qbc\" (UniqueName: \"kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc\") pod \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " Sep 07 10:35:32.247508 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.247406 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs\") pod \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " Sep 07 10:35:32.247508 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.247441 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") pod \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\" (UID: \"7a7aab48-9afe-4827-8ac7-f3e8329a234f\") " Sep 07 10:35:32.249678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.249648 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a7aab48-9afe-4827-8ac7-f3e8329a234f" (UID: "7a7aab48-9afe-4827-8ac7-f3e8329a234f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:35:32.249781 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.249703 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7a7aab48-9afe-4827-8ac7-f3e8329a234f" (UID: "7a7aab48-9afe-4827-8ac7-f3e8329a234f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:35:32.249781 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.249738 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc" (OuterVolumeSpecName: "kube-api-access-f4qbc") pod "7a7aab48-9afe-4827-8ac7-f3e8329a234f" (UID: "7a7aab48-9afe-4827-8ac7-f3e8329a234f"). InnerVolumeSpecName "kube-api-access-f4qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:35:32.349178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.349128 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-metrics-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:35:32.349178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.349171 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a7aab48-9afe-4827-8ac7-f3e8329a234f-webhook-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:35:32.349178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.349184 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qbc\" (UniqueName: \"kubernetes.io/projected/7a7aab48-9afe-4827-8ac7-f3e8329a234f-kube-api-access-f4qbc\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:35:32.898624 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.898563 2577 generic.go:356] "Generic (PLEG): container finished" podID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" containerID="b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad" exitCode=0 Sep 07 10:35:32.899073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.898621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-747c698f56-z2n7b" event={"ID":"7a7aab48-9afe-4827-8ac7-f3e8329a234f","Type":"ContainerDied","Data":"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad"} Sep 07 10:35:32.899073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.898643 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-747c698f56-z2n7b" Sep 07 10:35:32.899073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.898690 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-747c698f56-z2n7b" event={"ID":"7a7aab48-9afe-4827-8ac7-f3e8329a234f","Type":"ContainerDied","Data":"05e225af809df1fbf9d31761872bf4fecc2ec60f41e2d8868a82861505511cca"} Sep 07 10:35:32.899073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.898720 2577 scope.go:117] "RemoveContainer" containerID="b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad" Sep 07 10:35:32.908382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.908205 2577 scope.go:117] "RemoveContainer" containerID="b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad" Sep 07 10:35:32.908484 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:35:32.908467 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad\": container with ID starting with b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad not found: ID does not exist" containerID="b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad" Sep 07 10:35:32.908527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.908495 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad"} err="failed to get container status \"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad\": rpc error: code = NotFound desc = could not find container \"b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad\": container with ID starting with b526e6bc1f607f29d1f1a7688246c39b682c3b85102c411b1d33bce24afed8ad not found: ID does not exist" Sep 07 10:35:32.917245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.917213 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:32.920182 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:32.920159 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-747c698f56-z2n7b"] Sep 07 10:35:34.723459 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:34.723421 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" path="/var/lib/kubelet/pods/7a7aab48-9afe-4827-8ac7-f3e8329a234f/volumes" Sep 07 10:35:54.715020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.714981 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xhh2r"] Sep 07 10:35:54.715659 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.715551 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" containerName="manager" Sep 07 10:35:54.715659 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.715571 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" containerName="manager" Sep 07 10:35:54.715794 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.715722 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7aab48-9afe-4827-8ac7-f3e8329a234f" containerName="manager" Sep 07 10:35:54.724098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.724072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.726082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.726057 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 07 10:35:54.726196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.726112 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sb262\"" Sep 07 10:35:54.732388 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.732360 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xhh2r"] Sep 07 10:35:54.755806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.755773 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv"] Sep 07 10:35:54.759676 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.759660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.767875 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.767851 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv"] Sep 07 10:35:54.868352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.868320 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2297757b-3f4d-4aed-b1c6-1537057d0888-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.868538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.868370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.868538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.868396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmx9c\" (UniqueName: \"kubernetes.io/projected/2297757b-3f4d-4aed-b1c6-1537057d0888-kube-api-access-lmx9c\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.868538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.868491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbh5\" (UniqueName: \"kubernetes.io/projected/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-kube-api-access-6sbh5\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.969233 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.969197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmx9c\" (UniqueName: \"kubernetes.io/projected/2297757b-3f4d-4aed-b1c6-1537057d0888-kube-api-access-lmx9c\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.969434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.969246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbh5\" (UniqueName: \"kubernetes.io/projected/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-kube-api-access-6sbh5\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.969434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.969396 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2297757b-3f4d-4aed-b1c6-1537057d0888-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.969434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.969425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.970114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.970088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.970220 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.970175 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2297757b-3f4d-4aed-b1c6-1537057d0888-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:54.976646 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.976625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbh5\" (UniqueName: \"kubernetes.io/projected/f4328b5b-7c17-4f21-b3a5-5e890b59fbb1-kube-api-access-6sbh5\") pod \"payload-processing-5b54b7fbb7-xhh2r\" (UID: \"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:54.976789 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:54.976773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmx9c\" (UniqueName: \"kubernetes.io/projected/2297757b-3f4d-4aed-b1c6-1537057d0888-kube-api-access-lmx9c\") pod \"payload-pre-processing-59fbc47bf8-cp4dv\" (UID: \"2297757b-3f4d-4aed-b1c6-1537057d0888\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:55.035819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:55.035781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:55.070519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:55.070475 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:55.315406 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:55.315317 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xhh2r"] Sep 07 10:35:55.322781 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:35:55.322722 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4328b5b_7c17_4f21_b3a5_5e890b59fbb1.slice/crio-9efefb5d3ebb29a025edcbf0c6f94d145087f370c66b00c10c6ae6f7aaac86d1 WatchSource:0}: Error finding container 9efefb5d3ebb29a025edcbf0c6f94d145087f370c66b00c10c6ae6f7aaac86d1: Status 404 returned error can't find the container with id 9efefb5d3ebb29a025edcbf0c6f94d145087f370c66b00c10c6ae6f7aaac86d1 Sep 07 10:35:55.356213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:55.356188 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv"] Sep 07 10:35:55.358338 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:35:55.358311 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2297757b_3f4d_4aed_b1c6_1537057d0888.slice/crio-dce283b5b819f967df45344977c9191ffaebece71114168d39b63e4f57d9c2d2 WatchSource:0}: Error finding container dce283b5b819f967df45344977c9191ffaebece71114168d39b63e4f57d9c2d2: Status 404 returned error can't find the container with id dce283b5b819f967df45344977c9191ffaebece71114168d39b63e4f57d9c2d2 Sep 07 10:35:56.018386 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:56.018335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" event={"ID":"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1","Type":"ContainerStarted","Data":"9efefb5d3ebb29a025edcbf0c6f94d145087f370c66b00c10c6ae6f7aaac86d1"} Sep 07 10:35:56.019917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:56.019885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" event={"ID":"2297757b-3f4d-4aed-b1c6-1537057d0888","Type":"ContainerStarted","Data":"dce283b5b819f967df45344977c9191ffaebece71114168d39b63e4f57d9c2d2"} Sep 07 10:35:58.045261 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:58.045172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" event={"ID":"2297757b-3f4d-4aed-b1c6-1537057d0888","Type":"ContainerStarted","Data":"b32f4c40336bcbf404971815c423bbcd62c00d309107c06bfefec17e7b416c04"} Sep 07 10:35:58.045261 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:58.045248 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:35:58.061518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:58.061471 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" podStartSLOduration=1.680005553 podStartE2EDuration="4.061454416s" totalImagesPullingTime="2.381448863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:58.059702965 +0000 UTC m=+925.917392295" watchObservedRunningTime="2026-09-07 10:35:58.061454416 +0000 UTC m=+925.919143731" Sep 07 10:35:59.052545 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:59.052505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" event={"ID":"f4328b5b-7c17-4f21-b3a5-5e890b59fbb1","Type":"ContainerStarted","Data":"47b105e48a6fc523d0112d61d6edca09a6d725cc2eeb602ff69debde6387046b"} Sep 07 10:35:59.053083 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:59.052669 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:35:59.069086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:35:59.069035 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" podStartSLOduration=2.203567969 podStartE2EDuration="5.069003801s" totalImagesPullingTime="2.865435832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:35:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:35:59.068933693 +0000 UTC m=+926.926623011" watchObservedRunningTime="2026-09-07 10:35:59.069003801 +0000 UTC m=+926.926693118" Sep 07 10:36:04.082607 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.082501 2577 generic.go:356] "Generic (PLEG): container finished" podID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerID="95366c6c93825558d807dc30f28ddf1646c37b9fef4ae17f62a5c4f77909639c" exitCode=0 Sep 07 10:36:04.082998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.082574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" event={"ID":"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6","Type":"ContainerDied","Data":"95366c6c93825558d807dc30f28ddf1646c37b9fef4ae17f62a5c4f77909639c"} Sep 07 10:36:04.083040 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.082996 2577 scope.go:117] "RemoveContainer" containerID="95366c6c93825558d807dc30f28ddf1646c37b9fef4ae17f62a5c4f77909639c" Sep 07 10:36:04.084079 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.084053 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:36:04.084152 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.084135 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:36:04.084201 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:04.084186 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:36:05.089459 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:05.089424 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" event={"ID":"16b4cfa1-20f7-42a8-83a3-213f7be5cbb6","Type":"ContainerStarted","Data":"0d06c8447a3e8bf837d508c6077d860a6f24ec813cd31ee4b91ba4d8e203c93a"} Sep 07 10:36:05.548809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:05.548773 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:36:05.550004 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:05.549979 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-s2jpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 07 10:36:05.550080 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:05.550032 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podUID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 07 10:36:06.548498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:06.548456 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-s2jpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 07 10:36:06.548942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:06.548516 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podUID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 07 10:36:07.549060 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:07.549019 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-s2jpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 07 10:36:07.549487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:07.549084 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podUID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 07 10:36:08.548648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:08.548605 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-s2jpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 07 10:36:08.548859 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:08.548673 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podUID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 07 10:36:09.055329 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:09.055295 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-cp4dv" Sep 07 10:36:09.548793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:09.548754 2577 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-s2jpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 07 10:36:09.548983 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:09.548812 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" podUID="16b4cfa1-20f7-42a8-83a3-213f7be5cbb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 07 10:36:09.559165 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:09.559142 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:36:10.062003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:10.061971 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xhh2r" Sep 07 10:36:10.552403 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:10.552371 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:36:10.553454 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:10.553434 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-s2jpw" Sep 07 10:36:14.332293 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.332262 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv"] Sep 07 10:36:14.335107 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.335085 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.337290 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.337266 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 07 10:36:14.337724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.337701 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-v8l9q\"" Sep 07 10:36:14.337812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.337791 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 07 10:36:14.337812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.337805 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 07 10:36:14.345788 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.345769 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv"] Sep 07 10:36:14.382424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hb5\" (UniqueName: \"kubernetes.io/projected/79008255-a57f-421d-85b4-7089ed5ba0ab-kube-api-access-z2hb5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.382550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.382550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.382550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.382550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382520 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.382550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.382538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79008255-a57f-421d-85b4-7089ed5ba0ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.482974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hb5\" (UniqueName: \"kubernetes.io/projected/79008255-a57f-421d-85b4-7089ed5ba0ab-kube-api-access-z2hb5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483033 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483077 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483313 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483313 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79008255-a57f-421d-85b4-7089ed5ba0ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483448 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483553 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.483649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.483631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.485611 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.485557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79008255-a57f-421d-85b4-7089ed5ba0ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.486103 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.486080 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79008255-a57f-421d-85b4-7089ed5ba0ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.490493 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.490470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hb5\" (UniqueName: \"kubernetes.io/projected/79008255-a57f-421d-85b4-7089ed5ba0ab-kube-api-access-z2hb5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv\" (UID: \"79008255-a57f-421d-85b4-7089ed5ba0ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.646219 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.646144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:14.915043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.915020 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv"] Sep 07 10:36:14.917118 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:36:14.917092 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79008255_a57f_421d_85b4_7089ed5ba0ab.slice/crio-8a74736459624975f597bd2e4764c750b5b40b26c9a935387c82fc7afa347e3c WatchSource:0}: Error finding container 8a74736459624975f597bd2e4764c750b5b40b26c9a935387c82fc7afa347e3c: Status 404 returned error can't find the container with id 8a74736459624975f597bd2e4764c750b5b40b26c9a935387c82fc7afa347e3c Sep 07 10:36:14.918597 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:14.918560 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:36:15.156341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:15.156295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" event={"ID":"79008255-a57f-421d-85b4-7089ed5ba0ab","Type":"ContainerStarted","Data":"8a74736459624975f597bd2e4764c750b5b40b26c9a935387c82fc7afa347e3c"} Sep 07 10:36:16.163053 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:16.163018 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" event={"ID":"79008255-a57f-421d-85b4-7089ed5ba0ab","Type":"ContainerStarted","Data":"9750bf408c7cd48cd1ca89ef652b46ac357629e9f275d2e913ceb0dc04d80ddc"} Sep 07 10:36:16.181053 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:16.180932 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" podStartSLOduration=1.17996985 podStartE2EDuration="2.180914285s" totalImagesPullingTime="1.000944435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:16.179388188 +0000 UTC m=+944.037077494" watchObservedRunningTime="2026-09-07 10:36:16.180914285 +0000 UTC m=+944.038603602" Sep 07 10:36:24.646947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:24.646898 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:24.647420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:24.647010 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:24.659895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:24.659868 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:36:25.216730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:36:25.216698 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv" Sep 07 10:38:29.144070 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.143989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:38:29.147899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.147876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.149857 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.149836 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1ef687-openshift-default-dockercfg-cfsn9\"" Sep 07 10:38:29.158525 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.158502 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:38:29.209420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209431 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209452 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkdm\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209616 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.209904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.209808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.310723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.310688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.310895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.310737 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkdm\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.310895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.310861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.310993 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.310960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.310991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311428 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311622 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311552 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311622 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.311785 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.311767 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.313671 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.313648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.314045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.314024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.317913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.317892 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.318439 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.318413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkdm\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm\") pod \"e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.458537 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.458443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:29.736544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.736516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:38:29.739657 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:29.739632 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001cddd7_d6d0_459b_94d7_623348583a2f.slice/crio-e019881c9545fbb7260f8df8512e3e2043f621c557f2b2a5f2aaddb4e949538b WatchSource:0}: Error finding container e019881c9545fbb7260f8df8512e3e2043f621c557f2b2a5f2aaddb4e949538b: Status 404 returned error can't find the container with id e019881c9545fbb7260f8df8512e3e2043f621c557f2b2a5f2aaddb4e949538b Sep 07 10:38:29.742142 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.742113 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:29.742202 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.742173 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:29.742238 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.742205 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:29.812746 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.812718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerStarted","Data":"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f"} Sep 07 10:38:29.812839 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.812778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerStarted","Data":"e019881c9545fbb7260f8df8512e3e2043f621c557f2b2a5f2aaddb4e949538b"} Sep 07 10:38:29.833851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:29.833799 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podStartSLOduration=0.833780809 podStartE2EDuration="833.780809ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:29.831793397 +0000 UTC m=+1077.689482713" watchObservedRunningTime="2026-09-07 10:38:29.833780809 +0000 UTC m=+1077.691470125" Sep 07 10:38:30.459487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:30.459441 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:30.464570 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:30.464544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:30.818216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:30.818176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:30.819044 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:30.819025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:33.808502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.808468 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:38:33.812112 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.812096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:33.814294 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.814265 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1ef687-dockercfg-gsngc\"" Sep 07 10:38:33.814399 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.814297 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1ef687\"" Sep 07 10:38:33.821998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.821965 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:38:33.839713 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.839686 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:38:33.843221 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.843204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:33.856100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.856078 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:38:33.955540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.955507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qcr\" (UniqueName: \"kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:33.955868 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.955842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:33.956055 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.956030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrmq\" (UniqueName: \"kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:33.956190 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:33.956163 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.057119 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.057076 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qcr\" (UniqueName: \"kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.057328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.057146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.057328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.057202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrmq\" (UniqueName: \"kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.057328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.057294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.058269 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.058238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.058412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.058241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.065348 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.065265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qcr\" (UniqueName: \"kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr\") pod \"payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.065814 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.065788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrmq\" (UniqueName: \"kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq\") pod \"payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.125172 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.125138 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.154211 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.154112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.412009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.411980 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:38:34.413865 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:34.413835 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbd1546_ddaf_4529_9335_12a93b89b064.slice/crio-def61d1eabd41b7ff525551047fc4c8acb34d18fa01f64ebf4a571a6d83f4e58 WatchSource:0}: Error finding container def61d1eabd41b7ff525551047fc4c8acb34d18fa01f64ebf4a571a6d83f4e58: Status 404 returned error can't find the container with id def61d1eabd41b7ff525551047fc4c8acb34d18fa01f64ebf4a571a6d83f4e58 Sep 07 10:38:34.453466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.453442 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:38:34.461675 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:34.461646 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7cc9b2_0667_4ada_9887_bf6a43f6f633.slice/crio-27c7e5c871e5c8316990698b29625cbecba16406e0361230cd83a7911b18ab2a WatchSource:0}: Error finding container 27c7e5c871e5c8316990698b29625cbecba16406e0361230cd83a7911b18ab2a: Status 404 returned error can't find the container with id 27c7e5c871e5c8316990698b29625cbecba16406e0361230cd83a7911b18ab2a Sep 07 10:38:34.843188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.843147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" event={"ID":"2f7cc9b2-0667-4ada-9887-bf6a43f6f633","Type":"ContainerStarted","Data":"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5"} Sep 07 10:38:34.843698 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.843212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" event={"ID":"2f7cc9b2-0667-4ada-9887-bf6a43f6f633","Type":"ContainerStarted","Data":"27c7e5c871e5c8316990698b29625cbecba16406e0361230cd83a7911b18ab2a"} Sep 07 10:38:34.843698 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.843285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:34.845020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.844993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" event={"ID":"5bbd1546-ddaf-4529-9335-12a93b89b064","Type":"ContainerStarted","Data":"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87"} Sep 07 10:38:34.845158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.845043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" event={"ID":"5bbd1546-ddaf-4529-9335-12a93b89b064","Type":"ContainerStarted","Data":"def61d1eabd41b7ff525551047fc4c8acb34d18fa01f64ebf4a571a6d83f4e58"} Sep 07 10:38:34.845158 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.845099 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:34.859441 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.859398 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" podStartSLOduration=1.859385606 podStartE2EDuration="1.859385606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:34.857696205 +0000 UTC m=+1082.715385522" watchObservedRunningTime="2026-09-07 10:38:34.859385606 +0000 UTC m=+1082.717074922" Sep 07 10:38:34.873093 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:34.873054 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" podStartSLOduration=1.873043128 podStartE2EDuration="1.873043128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:34.871506235 +0000 UTC m=+1082.729195551" watchObservedRunningTime="2026-09-07 10:38:34.873043128 +0000 UTC m=+1082.730732443" Sep 07 10:38:37.173071 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.173036 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:38:37.178760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.178739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.180849 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.180828 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ea745819-openshift-default-dockercfg-xxrv9\"" Sep 07 10:38:37.189638 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.189614 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:38:37.288703 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.288703 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.288925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.288925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.288925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpt6\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.289044 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.289044 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.289044 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.288996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.289044 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.289012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.389991 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.389958 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390291 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpt6\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390538 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390679 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390699 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.390906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.390890 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.392841 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.392819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.393074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.393057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.397575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.397552 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.397812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.397792 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpt6\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6\") pod \"e2e-ait-ea745819-openshift-default-74d97d79b-6lr87\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.491658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.491621 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:37.774380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.774351 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:38:37.776569 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:37.776544 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714b0766_5095_44a3_b8e5_8539945520e0.slice/crio-7aee17270f8d4fbbac94a24ef2d5d743eb7a1f0621c06361b2724c429a8bd438 WatchSource:0}: Error finding container 7aee17270f8d4fbbac94a24ef2d5d743eb7a1f0621c06361b2724c429a8bd438: Status 404 returned error can't find the container with id 7aee17270f8d4fbbac94a24ef2d5d743eb7a1f0621c06361b2724c429a8bd438 Sep 07 10:38:37.778886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.778854 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:37.778995 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.778931 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:37.778995 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.778974 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:37.863644 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.863611 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerStarted","Data":"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb"} Sep 07 10:38:37.863775 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.863676 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerStarted","Data":"7aee17270f8d4fbbac94a24ef2d5d743eb7a1f0621c06361b2724c429a8bd438"} Sep 07 10:38:37.883876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:37.883820 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podStartSLOduration=0.88379843 podStartE2EDuration="883.79843ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:37.883424129 +0000 UTC m=+1085.741113458" watchObservedRunningTime="2026-09-07 10:38:37.88379843 +0000 UTC m=+1085.741487747" Sep 07 10:38:38.354156 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.354112 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:38:38.358615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.358571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.370341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.370318 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c5b29a-openshift-default-dockercfg-x2pvm\"" Sep 07 10:38:38.372432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.372407 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:38:38.492837 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.492802 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:38.496848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.496799 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:38:38.497220 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.497175 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:38:38.507041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507012 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507462 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjdr\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.507636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.507501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608568 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.608984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609099 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609099 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.608938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609099 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.609084 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjdr\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609327 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.609145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.609413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609712 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.609556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.609712 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.609652 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.611396 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.611375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.612020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.612000 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.617966 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.617927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.621526 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.621506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjdr\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr\") pod \"e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.669958 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.669936 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:38.956973 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.956948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:38:38.958675 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:38.958649 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddaed435_bb00_42b9_979e_847262f2b06b.slice/crio-68b0f8e0dfd05b56d30d082f2c570ec0fb2a39a031729cf60fcf403aac19d62d WatchSource:0}: Error finding container 68b0f8e0dfd05b56d30d082f2c570ec0fb2a39a031729cf60fcf403aac19d62d: Status 404 returned error can't find the container with id 68b0f8e0dfd05b56d30d082f2c570ec0fb2a39a031729cf60fcf403aac19d62d Sep 07 10:38:38.960943 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.960912 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:38.961005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.960976 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:38.961038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:38.961006 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:39.496746 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.496715 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:39.877412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.877315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerStarted","Data":"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e"} Sep 07 10:38:39.877412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.877375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerStarted","Data":"68b0f8e0dfd05b56d30d082f2c570ec0fb2a39a031729cf60fcf403aac19d62d"} Sep 07 10:38:39.877686 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.877576 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:39.878734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.878711 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:39.898076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:39.898029 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podStartSLOduration=1.898015572 podStartE2EDuration="1.898015572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:39.897026107 +0000 UTC m=+1087.754715433" watchObservedRunningTime="2026-09-07 10:38:39.898015572 +0000 UTC m=+1087.755704888" Sep 07 10:38:40.670908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:40.670866 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:40.675910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:40.675885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:40.883993 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:40.883921 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:40.885236 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:40.885213 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:38:41.047441 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.047411 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:38:41.051203 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.051181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.053213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.053195 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea745819\"" Sep 07 10:38:41.053318 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.053284 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea745819-dockercfg-lsxfl\"" Sep 07 10:38:41.062903 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.062881 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:38:41.143105 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.143069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfb4x\" (UniqueName: \"kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.143263 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.143113 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.244381 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.244339 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfb4x\" (UniqueName: \"kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.244381 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.244379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.245208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.245183 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.252298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.252277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfb4x\" (UniqueName: \"kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x\") pod \"payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.363685 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.363596 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.663683 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.663617 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:38:41.666042 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:38:41.666002 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26e4b92_56ac_49d4_a707_db7c7f6235d4.slice/crio-b4c90904fb6c5db16ce16b710a7937b70c4d5bc6db507a76c41fd6af7c0c49b9 WatchSource:0}: Error finding container b4c90904fb6c5db16ce16b710a7937b70c4d5bc6db507a76c41fd6af7c0c49b9: Status 404 returned error can't find the container with id b4c90904fb6c5db16ce16b710a7937b70c4d5bc6db507a76c41fd6af7c0c49b9 Sep 07 10:38:41.891735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.891691 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" event={"ID":"a26e4b92-56ac-49d4-a707-db7c7f6235d4","Type":"ContainerStarted","Data":"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a"} Sep 07 10:38:41.892208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.892070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:41.892208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.892111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" event={"ID":"a26e4b92-56ac-49d4-a707-db7c7f6235d4","Type":"ContainerStarted","Data":"b4c90904fb6c5db16ce16b710a7937b70c4d5bc6db507a76c41fd6af7c0c49b9"} Sep 07 10:38:41.909665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:41.909603 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" podStartSLOduration=0.909562928 podStartE2EDuration="909.562928ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:41.907903147 +0000 UTC m=+1089.765592465" watchObservedRunningTime="2026-09-07 10:38:41.909562928 +0000 UTC m=+1089.767252244" Sep 07 10:38:45.855651 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:45.855617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:38:45.856246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:45.855702 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:38:46.859774 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:46.859736 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:38:46.860218 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:46.859804 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:38:48.947286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.947250 2577 generic.go:356] "Generic (PLEG): container finished" podID="001cddd7-d6d0-459b-94d7-623348583a2f" containerID="cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f" exitCode=0 Sep 07 10:38:48.947833 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.947324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerDied","Data":"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f"} Sep 07 10:38:48.947908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.947876 2577 scope.go:117] "RemoveContainer" containerID="cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f" Sep 07 10:38:48.949024 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.948997 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:48.949086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.949065 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:48.949123 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:48.949095 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:49.959174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:49.959133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerStarted","Data":"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e"} Sep 07 10:38:50.460657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:50.459069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:50.460657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:50.459199 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:38:50.460657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:50.459245 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 07 10:38:51.459265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:51.459220 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:38:51.459707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:51.459292 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 07 10:38:52.459762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:52.459726 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:38:52.460110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:52.459795 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 07 10:38:53.459410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:53.459362 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:38:53.459613 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:53.459455 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 07 10:38:53.909806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:53.909769 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:38:54.459282 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:54.459242 2577 patch_prober.go:28] interesting pod/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Sep 07 10:38:54.459490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:54.459305 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Sep 07 10:38:55.540007 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:55.538918 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:55.540007 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:55.539914 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:55.544039 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:55.544014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:38:55.920133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:55.919724 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:38:55.920133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:55.919791 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:38:57.001173 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.001146 2577 generic.go:356] "Generic (PLEG): container finished" podID="714b0766-5095-44a3-b8e5-8539945520e0" containerID="bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb" exitCode=0 Sep 07 10:38:57.001514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.001219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerDied","Data":"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb"} Sep 07 10:38:57.001749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.001736 2577 scope.go:117] "RemoveContainer" containerID="bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb" Sep 07 10:38:57.002760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.002733 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:57.002867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.002813 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:57.002867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:57.002860 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:38:58.010504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:58.010462 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerStarted","Data":"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262"} Sep 07 10:38:58.491845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:58.491816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:38:58.492014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:58.491995 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:38:58.492061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:58.492042 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:38:59.492026 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:59.491991 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:38:59.492372 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:38:59.492049 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:39:00.492819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:00.492772 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:39:00.493224 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:00.492857 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:39:01.492994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:01.492956 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:39:01.493398 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:01.493030 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:39:02.492271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:02.492234 2577 patch_prober.go:28] interesting pod/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 07 10:39:02.492461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:02.492291 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 07 10:39:03.517393 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:03.517355 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:39:03.517994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:03.517969 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:39:03.519747 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:03.519720 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:39:04.057939 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.057900 2577 generic.go:356] "Generic (PLEG): container finished" podID="ddaed435-bb00-42b9-979e-847262f2b06b" containerID="334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e" exitCode=0 Sep 07 10:39:04.058132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.057983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerDied","Data":"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e"} Sep 07 10:39:04.058511 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.058498 2577 scope.go:117] "RemoveContainer" containerID="334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e" Sep 07 10:39:04.059620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.059571 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:04.059683 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.059655 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:04.059719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:04.059685 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:05.064940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.064906 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerStarted","Data":"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4"} Sep 07 10:39:05.559369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.559335 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:39:05.563113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.563097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.568769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.568749 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b4061f\"" Sep 07 10:39:05.568867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.568851 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b4061f-dockercfg-cn9kl\"" Sep 07 10:39:05.575491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.575472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:39:05.650463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.650418 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.650675 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.650601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznvr\" (UniqueName: \"kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.670072 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.670037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:39:05.670432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.670396 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:39:05.670499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.670472 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 07 10:39:05.752249 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.752203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.752453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.752308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznvr\" (UniqueName: \"kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.753005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.752977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.762968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.762943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznvr\" (UniqueName: \"kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr\") pod \"payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:05.873069 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:05.872976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:06.167503 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:06.167481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:39:06.169908 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:39:06.169881 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04037e8_d806_4f47_a1d5_65796dd553e4.slice/crio-a63c08011aca5a91f2f5a545cbc2170d5ee07977a57d875b8fd74e12a17669a7 WatchSource:0}: Error finding container a63c08011aca5a91f2f5a545cbc2170d5ee07977a57d875b8fd74e12a17669a7: Status 404 returned error can't find the container with id a63c08011aca5a91f2f5a545cbc2170d5ee07977a57d875b8fd74e12a17669a7 Sep 07 10:39:06.670271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:06.670235 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:39:06.670490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:06.670291 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 07 10:39:07.077527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.077491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" event={"ID":"e04037e8-d806-4f47-a1d5-65796dd553e4","Type":"ContainerStarted","Data":"947254ec46c7c63aeabf3284ef77fc84b0e496666db4b69bbc308c4c7cbece5c"} Sep 07 10:39:07.077774 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.077553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:07.077774 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.077570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" event={"ID":"e04037e8-d806-4f47-a1d5-65796dd553e4","Type":"ContainerStarted","Data":"a63c08011aca5a91f2f5a545cbc2170d5ee07977a57d875b8fd74e12a17669a7"} Sep 07 10:39:07.094157 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.094104 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" podStartSLOduration=2.094084918 podStartE2EDuration="2.094084918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:07.093428338 +0000 UTC m=+1114.951117656" watchObservedRunningTime="2026-09-07 10:39:07.094084918 +0000 UTC m=+1114.951774238" Sep 07 10:39:07.671001 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.670969 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:39:07.671419 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:07.671025 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 07 10:39:08.670484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:08.670320 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:39:08.670484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:08.670388 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 07 10:39:09.676090 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:09.673021 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 07 10:39:09.676090 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:09.673121 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 07 10:39:10.800073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:10.800039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:39:10.800519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:10.800369 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:39:10.801741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:10.801721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:39:18.085325 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:18.085293 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:39:21.238741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.238703 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:39:21.239166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.238987 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" podUID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" containerName="payload-processing" containerID="cri-o://c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a" gracePeriod=30 Sep 07 10:39:21.501419 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.501363 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:39:21.547133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.547098 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:39:21.547389 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.547339 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" containerID="cri-o://a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262" gracePeriod=30 Sep 07 10:39:21.620925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.620897 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfb4x\" (UniqueName: \"kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x\") pod \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " Sep 07 10:39:21.621068 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.620951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume\") pod \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\" (UID: \"a26e4b92-56ac-49d4-a707-db7c7f6235d4\") " Sep 07 10:39:21.621338 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.621314 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a26e4b92-56ac-49d4-a707-db7c7f6235d4" (UID: "a26e4b92-56ac-49d4-a707-db7c7f6235d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:21.623160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.623140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x" (OuterVolumeSpecName: "kube-api-access-kfb4x") pod "a26e4b92-56ac-49d4-a707-db7c7f6235d4" (UID: "a26e4b92-56ac-49d4-a707-db7c7f6235d4"). InnerVolumeSpecName "kube-api-access-kfb4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:39:21.721748 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.721720 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfb4x\" (UniqueName: \"kubernetes.io/projected/a26e4b92-56ac-49d4-a707-db7c7f6235d4-kube-api-access-kfb4x\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:21.721748 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:21.721745 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26e4b92-56ac-49d4-a707-db7c7f6235d4-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:22.161854 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.161817 2577 generic.go:356] "Generic (PLEG): container finished" podID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" containerID="c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a" exitCode=0 Sep 07 10:39:22.162055 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.161873 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" Sep 07 10:39:22.162055 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.161891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" event={"ID":"a26e4b92-56ac-49d4-a707-db7c7f6235d4","Type":"ContainerDied","Data":"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a"} Sep 07 10:39:22.162055 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.161934 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n" event={"ID":"a26e4b92-56ac-49d4-a707-db7c7f6235d4","Type":"ContainerDied","Data":"b4c90904fb6c5db16ce16b710a7937b70c4d5bc6db507a76c41fd6af7c0c49b9"} Sep 07 10:39:22.162055 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.161955 2577 scope.go:117] "RemoveContainer" containerID="c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a" Sep 07 10:39:22.171684 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.171668 2577 scope.go:117] "RemoveContainer" containerID="c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a" Sep 07 10:39:22.171956 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:39:22.171937 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a\": container with ID starting with c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a not found: ID does not exist" containerID="c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a" Sep 07 10:39:22.172015 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.171963 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a"} err="failed to get container status \"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a\": rpc error: code = NotFound desc = could not find container \"c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a\": container with ID starting with c729d9b9647bc6c4a80f7c3a1489e49246bf72f17c7ad7226293db40ad2f788a not found: ID does not exist" Sep 07 10:39:22.185271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.185248 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:39:22.188825 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.188803 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea745819-ff6cf9f4c-kqn5n"] Sep 07 10:39:22.603304 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.603274 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:39:22.603973 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.603948 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" containerName="payload-processing" Sep 07 10:39:22.603973 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.603973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" containerName="payload-processing" Sep 07 10:39:22.604153 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.604121 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" containerName="payload-processing" Sep 07 10:39:22.608846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.608816 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.610914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.610888 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6cd248c6-gw-openshift-default-dockercfg-wj4wz\"" Sep 07 10:39:22.619446 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.619423 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.738162 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.735991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4xv\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.744260 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.744228 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a26e4b92-56ac-49d4-a707-db7c7f6235d4" path="/var/lib/kubelet/pods/a26e4b92-56ac-49d4-a707-db7c7f6235d4/volumes" Sep 07 10:39:22.837280 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4xv\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837280 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837356 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837797 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837797 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837797 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837887 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.837952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.837916 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.838041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.838007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.838041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.838014 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.838304 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.838287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.839964 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.839943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.840218 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.840197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.849437 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.849410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.849843 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.849824 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4xv\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv\") pod \"e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:22.920909 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:22.920829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:23.200427 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:23.200401 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:39:23.201981 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:39:23.201955 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7dae77_7a71_42b7_b625_903bdae5dc7a.slice/crio-b54615e56f06db3dee8487e8adae8eaf3580a54337e7844d00905ea27817f8f2 WatchSource:0}: Error finding container b54615e56f06db3dee8487e8adae8eaf3580a54337e7844d00905ea27817f8f2: Status 404 returned error can't find the container with id b54615e56f06db3dee8487e8adae8eaf3580a54337e7844d00905ea27817f8f2 Sep 07 10:39:23.204047 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:23.204020 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:23.204125 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:23.204078 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:23.204125 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:23.204121 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:24.175410 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:24.175360 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerStarted","Data":"c608dd08e7dd4608060ebdc4af577fc35a31838a414ce6f8b3e60818fa2218af"} Sep 07 10:39:24.175794 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:24.175417 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerStarted","Data":"b54615e56f06db3dee8487e8adae8eaf3580a54337e7844d00905ea27817f8f2"} Sep 07 10:39:24.194326 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:24.194285 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podStartSLOduration=2.194269885 podStartE2EDuration="2.194269885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:24.19330567 +0000 UTC m=+1132.050994979" watchObservedRunningTime="2026-09-07 10:39:24.194269885 +0000 UTC m=+1132.051959200" Sep 07 10:39:24.921972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:24.921934 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:24.927177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:24.927151 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:25.182150 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:25.182071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:25.183056 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:25.183037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:26.817968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.817910 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:39:26.880401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880379 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880529 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880426 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880529 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880447 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880529 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880516 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880640 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880692 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:26.880933 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880756 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmpt6\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880933 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880808 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket\") pod \"714b0766-5095-44a3-b8e5-8539945520e0\" (UID: \"714b0766-5095-44a3-b8e5-8539945520e0\") " Sep 07 10:39:26.880933 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880856 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:26.881092 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.880989 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:26.881092 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881008 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:26.881206 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881163 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:26.881541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881515 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/714b0766-5095-44a3-b8e5-8539945520e0-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.881678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881544 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.881678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881560 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.881678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881574 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.881678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.881604 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.882840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.882816 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:39:26.882998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.882980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:39:26.883052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.883000 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:39:26.883221 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.883208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6" (OuterVolumeSpecName: "kube-api-access-cmpt6") pod "714b0766-5095-44a3-b8e5-8539945520e0" (UID: "714b0766-5095-44a3-b8e5-8539945520e0"). InnerVolumeSpecName "kube-api-access-cmpt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:39:26.982502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.982480 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpt6\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-kube-api-access-cmpt6\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.982502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.982499 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/714b0766-5095-44a3-b8e5-8539945520e0-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.982648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.982509 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/714b0766-5095-44a3-b8e5-8539945520e0-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:26.982648 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:26.982518 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/714b0766-5095-44a3-b8e5-8539945520e0-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:27.194149 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.194065 2577 generic.go:356] "Generic (PLEG): container finished" podID="714b0766-5095-44a3-b8e5-8539945520e0" containerID="a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262" exitCode=0 Sep 07 10:39:27.194149 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.194111 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" Sep 07 10:39:27.194149 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.194137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerDied","Data":"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262"} Sep 07 10:39:27.194426 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.194192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87" event={"ID":"714b0766-5095-44a3-b8e5-8539945520e0","Type":"ContainerDied","Data":"7aee17270f8d4fbbac94a24ef2d5d743eb7a1f0621c06361b2724c429a8bd438"} Sep 07 10:39:27.194426 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.194214 2577 scope.go:117] "RemoveContainer" containerID="a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262" Sep 07 10:39:27.207842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.207820 2577 scope.go:117] "RemoveContainer" containerID="bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb" Sep 07 10:39:27.216470 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.216453 2577 scope.go:117] "RemoveContainer" containerID="a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262" Sep 07 10:39:27.216801 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:39:27.216776 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262\": container with ID starting with a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262 not found: ID does not exist" containerID="a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262" Sep 07 10:39:27.216871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.216815 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262"} err="failed to get container status \"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262\": rpc error: code = NotFound desc = could not find container \"a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262\": container with ID starting with a2f8536833c2742749308b2d4ed03428b4125f3742d593bc28b7d3ccbae8a262 not found: ID does not exist" Sep 07 10:39:27.216871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.216841 2577 scope.go:117] "RemoveContainer" containerID="bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb" Sep 07 10:39:27.217191 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:39:27.217165 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb\": container with ID starting with bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb not found: ID does not exist" containerID="bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb" Sep 07 10:39:27.217261 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.217200 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb"} err="failed to get container status \"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb\": rpc error: code = NotFound desc = could not find container \"bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb\": container with ID starting with bd5d32c531d6bd2c548e01f5ff06577e522be3b281339549fe20e4a348f5bcbb not found: ID does not exist" Sep 07 10:39:27.221941 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.221919 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:39:27.225221 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.225201 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ea745819-openshift-default-74d97d79b-6lr87"] Sep 07 10:39:27.367939 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.367910 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:39:27.368353 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.368341 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:27.368397 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.368355 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:27.368441 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.368432 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:27.368477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.368445 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:27.373307 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.373282 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.375686 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.375671 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6cd248c6-dockercfg-wbsrz\"" Sep 07 10:39:27.375811 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.375733 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6cd248c6\"" Sep 07 10:39:27.378824 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.378802 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:39:27.487969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.487936 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dxf\" (UniqueName: \"kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.488134 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.488076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.589105 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.589064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.589275 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.589151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dxf\" (UniqueName: \"kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.589816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.589797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.604347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.604320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dxf\" (UniqueName: \"kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf\") pod \"payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.684920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.684891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:27.970768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:27.970745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:39:27.972465 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:39:27.972442 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e99faa2_9f67_40f0_a8b9_6df7cab9dcd8.slice/crio-792e896f542d4543a968602a49a5f7dbff39ba95c3554222be4ad3fb50fac32d WatchSource:0}: Error finding container 792e896f542d4543a968602a49a5f7dbff39ba95c3554222be4ad3fb50fac32d: Status 404 returned error can't find the container with id 792e896f542d4543a968602a49a5f7dbff39ba95c3554222be4ad3fb50fac32d Sep 07 10:39:28.201777 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:28.201734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" event={"ID":"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8","Type":"ContainerStarted","Data":"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b"} Sep 07 10:39:28.201967 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:28.201791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" event={"ID":"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8","Type":"ContainerStarted","Data":"792e896f542d4543a968602a49a5f7dbff39ba95c3554222be4ad3fb50fac32d"} Sep 07 10:39:28.201967 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:28.201826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:28.216791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:28.216728 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" podStartSLOduration=1.216708053 podStartE2EDuration="1.216708053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:28.215356405 +0000 UTC m=+1136.073045720" watchObservedRunningTime="2026-09-07 10:39:28.216708053 +0000 UTC m=+1136.074397372" Sep 07 10:39:28.724409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:28.724371 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714b0766-5095-44a3-b8e5-8539945520e0" path="/var/lib/kubelet/pods/714b0766-5095-44a3-b8e5-8539945520e0/volumes" Sep 07 10:39:39.211996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:39.211953 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:39:41.223749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:41.223705 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:39:41.224234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:41.223762 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:39:43.292350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.292313 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerID="c608dd08e7dd4608060ebdc4af577fc35a31838a414ce6f8b3e60818fa2218af" exitCode=0 Sep 07 10:39:43.292764 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.292391 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerDied","Data":"c608dd08e7dd4608060ebdc4af577fc35a31838a414ce6f8b3e60818fa2218af"} Sep 07 10:39:43.292808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.292781 2577 scope.go:117] "RemoveContainer" containerID="c608dd08e7dd4608060ebdc4af577fc35a31838a414ce6f8b3e60818fa2218af" Sep 07 10:39:43.293873 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.293842 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:43.293976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.293915 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:43.293976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:43.293949 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:39:44.300078 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:44.300042 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerStarted","Data":"b7bc5420ec8cdb91dcff93877228449ab39502d2cebc4e80a6070f397bddad11"} Sep 07 10:39:44.921914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:44.921883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:44.922290 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:44.922203 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 07 10:39:44.922460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:44.922436 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 07 10:39:45.921540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:45.921489 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 07 10:39:45.922019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:45.921567 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 07 10:39:46.922188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:46.922141 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 07 10:39:46.922707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:46.922210 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 07 10:39:47.922030 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:47.921980 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 07 10:39:47.922255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:47.922067 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 07 10:39:48.921630 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:48.921561 2577 patch_prober.go:28] interesting pod/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 07 10:39:48.921834 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:48.921654 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 07 10:39:49.925235 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:49.925207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:49.925707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:49.925481 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:49.926169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:49.926153 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:39:50.765033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.765002 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq"] Sep 07 10:39:50.765487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.765456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:50.765487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.765473 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="714b0766-5095-44a3-b8e5-8539945520e0" containerName="istio-proxy" Sep 07 10:39:50.768919 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.768901 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:50.770972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.770952 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:39:50.771085 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.770954 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:39:50.776012 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.775987 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq"] Sep 07 10:39:50.826887 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.826859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq\" (UID: \"1619146b-855c-4021-8d16-a0889f0f3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:50.927649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.927620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq\" (UID: \"1619146b-855c-4021-8d16-a0889f0f3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:50.928215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:50.928197 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq\" (UID: \"1619146b-855c-4021-8d16-a0889f0f3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:51.083206 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:51.083129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:51.356700 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:51.356675 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq"] Sep 07 10:39:51.358542 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:39:51.358510 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1619146b_855c_4021_8d16_a0889f0f3ef3.slice/crio-55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d WatchSource:0}: Error finding container 55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d: Status 404 returned error can't find the container with id 55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d Sep 07 10:39:52.360008 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:52.359963 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" event={"ID":"1619146b-855c-4021-8d16-a0889f0f3ef3","Type":"ContainerStarted","Data":"55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d"} Sep 07 10:39:54.373409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:54.373373 2577 generic.go:356] "Generic (PLEG): container finished" podID="1619146b-855c-4021-8d16-a0889f0f3ef3" containerID="ce419d38d940b9691fe04414de3f5758bc913c90feb7b464259eaeb099bcf78d" exitCode=0 Sep 07 10:39:54.373770 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:54.373451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" event={"ID":"1619146b-855c-4021-8d16-a0889f0f3ef3","Type":"ContainerDied","Data":"ce419d38d940b9691fe04414de3f5758bc913c90feb7b464259eaeb099bcf78d"} Sep 07 10:39:55.514957 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:55.514933 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:55.576714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:55.576687 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca\") pod \"1619146b-855c-4021-8d16-a0889f0f3ef3\" (UID: \"1619146b-855c-4021-8d16-a0889f0f3ef3\") " Sep 07 10:39:55.577033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:55.577011 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1619146b-855c-4021-8d16-a0889f0f3ef3" (UID: "1619146b-855c-4021-8d16-a0889f0f3ef3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:39:55.678306 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:55.678213 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1619146b-855c-4021-8d16-a0889f0f3ef3-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:39:56.389257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:56.389214 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" Sep 07 10:39:56.389453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:56.389213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq" event={"ID":"1619146b-855c-4021-8d16-a0889f0f3ef3","Type":"ContainerDied","Data":"55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d"} Sep 07 10:39:56.389453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:39:56.389393 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55ae4700007905b29f3d024d1b2c44a617f2926f5c7f011af26b3110f0a12e9d" Sep 07 10:40:06.965611 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:06.955914 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:40:06.965611 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:06.956181 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" podUID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" containerName="payload-pre-processing" containerID="cri-o://a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b" gracePeriod=30 Sep 07 10:40:07.203672 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.203549 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq"] Sep 07 10:40:07.230250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.230219 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6cd248c6-5vlgq"] Sep 07 10:40:07.349187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.348888 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:40:07.364344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.364309 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:40:07.364865 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.364808 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" containerID="cri-o://b7bc5420ec8cdb91dcff93877228449ab39502d2cebc4e80a6070f397bddad11" gracePeriod=30 Sep 07 10:40:07.414187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.414144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88dxf\" (UniqueName: \"kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf\") pod \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " Sep 07 10:40:07.414923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.414899 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume\") pod \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\" (UID: \"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8\") " Sep 07 10:40:07.419883 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.419773 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" (UID: "4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:07.420392 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.420370 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:07.442606 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.436040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf" (OuterVolumeSpecName: "kube-api-access-88dxf") pod "4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" (UID: "4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8"). InnerVolumeSpecName "kube-api-access-88dxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:07.470888 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.470841 2577 generic.go:356] "Generic (PLEG): container finished" podID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" containerID="a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b" exitCode=0 Sep 07 10:40:07.471405 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.471228 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" Sep 07 10:40:07.472369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.471252 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" event={"ID":"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8","Type":"ContainerDied","Data":"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b"} Sep 07 10:40:07.472606 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.472543 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l" event={"ID":"4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8","Type":"ContainerDied","Data":"792e896f542d4543a968602a49a5f7dbff39ba95c3554222be4ad3fb50fac32d"} Sep 07 10:40:07.472846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.472767 2577 scope.go:117] "RemoveContainer" containerID="a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b" Sep 07 10:40:07.513131 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.513103 2577 scope.go:117] "RemoveContainer" containerID="a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b" Sep 07 10:40:07.513634 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:40:07.513558 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b\": container with ID starting with a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b not found: ID does not exist" containerID="a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b" Sep 07 10:40:07.513741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.513646 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b"} err="failed to get container status \"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b\": rpc error: code = NotFound desc = could not find container \"a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b\": container with ID starting with a07cb785bebd8f3f7677396cf177b6ee95cfc26c5faabddca4782807c291d26b not found: ID does not exist" Sep 07 10:40:07.523047 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.522246 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dxf\" (UniqueName: \"kubernetes.io/projected/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8-kube-api-access-88dxf\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:07.525567 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.525478 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:40:07.530645 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:07.529664 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6cd248c6-5cf66456b-vvs9l"] Sep 07 10:40:08.509933 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.509898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:08.510405 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510391 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" containerName="payload-pre-processing" Sep 07 10:40:08.510449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510406 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" containerName="payload-pre-processing" Sep 07 10:40:08.510449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510435 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1619146b-855c-4021-8d16-a0889f0f3ef3" containerName="revoke-keys" Sep 07 10:40:08.510449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510444 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1619146b-855c-4021-8d16-a0889f0f3ef3" containerName="revoke-keys" Sep 07 10:40:08.510538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510530 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" containerName="payload-pre-processing" Sep 07 10:40:08.510569 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.510544 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1619146b-855c-4021-8d16-a0889f0f3ef3" containerName="revoke-keys" Sep 07 10:40:08.515871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.515850 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.518185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.518170 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fc9185fe-openshift-default-dockercfg-qch47\"" Sep 07 10:40:08.527327 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.527303 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:08.632063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.632021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.632395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.632372 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.632623 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.632606 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.632759 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.632744 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzgv\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.632915 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.632899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.633077 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.633061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.633354 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.633337 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.633497 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.633483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.633660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.633644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.723375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.723342 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1619146b-855c-4021-8d16-a0889f0f3ef3" path="/var/lib/kubelet/pods/1619146b-855c-4021-8d16-a0889f0f3ef3/volumes" Sep 07 10:40:08.723763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.723749 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8" path="/var/lib/kubelet/pods/4e99faa2-9f67-40f0-a8b9-6df7cab9dcd8/volumes" Sep 07 10:40:08.735040 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzgv\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735571 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735542 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735690 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735819 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.735858 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.735822 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.737754 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.737733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.737867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.737852 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.742618 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.742574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzgv\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.743042 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.743022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token\") pod \"e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:08.827989 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:08.827928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:09.107242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.107219 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:09.109081 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:09.109036 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b11de89_deba_4c46_ae2a_a48f8d8b4ee2.slice/crio-df081ef41eb84ca3fa6085b22822b54555b1042d73442d2cf81870701a602fb8 WatchSource:0}: Error finding container df081ef41eb84ca3fa6085b22822b54555b1042d73442d2cf81870701a602fb8: Status 404 returned error can't find the container with id df081ef41eb84ca3fa6085b22822b54555b1042d73442d2cf81870701a602fb8 Sep 07 10:40:09.111674 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.111635 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:09.111752 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.111727 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:09.111790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.111777 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:09.487532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.487493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerStarted","Data":"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c"} Sep 07 10:40:09.487767 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.487555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerStarted","Data":"df081ef41eb84ca3fa6085b22822b54555b1042d73442d2cf81870701a602fb8"} Sep 07 10:40:09.505436 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.505394 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podStartSLOduration=1.505379316 podStartE2EDuration="1.505379316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:09.503969716 +0000 UTC m=+1177.361659034" watchObservedRunningTime="2026-09-07 10:40:09.505379316 +0000 UTC m=+1177.363068632" Sep 07 10:40:09.828222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:09.828129 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:10.833270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:10.833242 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:11.499054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:11.499024 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:11.500100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:11.500082 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:12.506152 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.506119 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerID="b7bc5420ec8cdb91dcff93877228449ab39502d2cebc4e80a6070f397bddad11" exitCode=0 Sep 07 10:40:12.506620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.506200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerDied","Data":"b7bc5420ec8cdb91dcff93877228449ab39502d2cebc4e80a6070f397bddad11"} Sep 07 10:40:12.506620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.506281 2577 scope.go:117] "RemoveContainer" containerID="c608dd08e7dd4608060ebdc4af577fc35a31838a414ce6f8b3e60818fa2218af" Sep 07 10:40:12.624328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.624308 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:40:12.778957 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.778929 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779107 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779014 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779107 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779107 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779299 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779239 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp4xv\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779353 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:12.779353 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779303 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779353 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779330 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:12.779527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779354 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:12.779527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779380 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert\") pod \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\" (UID: \"9f7dae77-7a71-42b7-b625-903bdae5dc7a\") " Sep 07 10:40:12.779882 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779835 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:12.780006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779974 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.780006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.779994 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.780136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.780009 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.780136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.780023 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.780245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.780146 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:12.781384 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.781359 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:12.781472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.781388 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:40:12.781472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.781442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:12.781556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.781475 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv" (OuterVolumeSpecName: "kube-api-access-xp4xv") pod "9f7dae77-7a71-42b7-b625-903bdae5dc7a" (UID: "9f7dae77-7a71-42b7-b625-903bdae5dc7a"). InnerVolumeSpecName "kube-api-access-xp4xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:12.881002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.880971 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.881002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.880996 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.881002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.881005 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4xv\" (UniqueName: \"kubernetes.io/projected/9f7dae77-7a71-42b7-b625-903bdae5dc7a-kube-api-access-xp4xv\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.881216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.881016 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.881216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.881025 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f7dae77-7a71-42b7-b625-903bdae5dc7a-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:12.913936 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.913908 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:12.914355 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914342 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.914404 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914357 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.914404 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914370 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.914404 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914375 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.914498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914448 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.914498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.914456 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" containerName="istio-proxy" Sep 07 10:40:12.917843 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.917827 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:12.921897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.921877 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc9185fe-dockercfg-kl4f6\"" Sep 07 10:40:12.921987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.921899 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc9185fe\"" Sep 07 10:40:12.933234 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:12.933208 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:13.083119 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.083028 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.083119 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.083114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv7d\" (UniqueName: \"kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.184756 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.184717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.184952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.184794 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv7d\" (UniqueName: \"kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.185432 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.185410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.193155 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.193130 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv7d\" (UniqueName: \"kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d\") pod \"payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.228212 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.228186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:13.513549 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.513518 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" Sep 07 10:40:13.514019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.513559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg" event={"ID":"9f7dae77-7a71-42b7-b625-903bdae5dc7a","Type":"ContainerDied","Data":"b54615e56f06db3dee8487e8adae8eaf3580a54337e7844d00905ea27817f8f2"} Sep 07 10:40:13.514019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.513643 2577 scope.go:117] "RemoveContainer" containerID="b7bc5420ec8cdb91dcff93877228449ab39502d2cebc4e80a6070f397bddad11" Sep 07 10:40:13.519416 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.519392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:13.526300 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:13.526278 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65eb903_702a_49e8_a7bf_c2b2d80d6da2.slice/crio-92711be8dab398b337d36eb98cbd1fcf5cdb91f41d3f287831cb5b387600eef6 WatchSource:0}: Error finding container 92711be8dab398b337d36eb98cbd1fcf5cdb91f41d3f287831cb5b387600eef6: Status 404 returned error can't find the container with id 92711be8dab398b337d36eb98cbd1fcf5cdb91f41d3f287831cb5b387600eef6 Sep 07 10:40:13.549216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.549191 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:40:13.552726 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:13.552703 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6cd248c6-gw-openshift-default-8f7dff656-ml8fg"] Sep 07 10:40:14.522049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:14.522020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" event={"ID":"f65eb903-702a-49e8-a7bf-c2b2d80d6da2","Type":"ContainerStarted","Data":"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e"} Sep 07 10:40:14.522455 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:14.522066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" event={"ID":"f65eb903-702a-49e8-a7bf-c2b2d80d6da2","Type":"ContainerStarted","Data":"92711be8dab398b337d36eb98cbd1fcf5cdb91f41d3f287831cb5b387600eef6"} Sep 07 10:40:14.522455 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:14.522116 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:14.539068 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:14.539023 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" podStartSLOduration=2.5390103809999998 podStartE2EDuration="2.539010381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:14.537896293 +0000 UTC m=+1182.395585610" watchObservedRunningTime="2026-09-07 10:40:14.539010381 +0000 UTC m=+1182.396699720" Sep 07 10:40:14.723496 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:14.723460 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7dae77-7a71-42b7-b625-903bdae5dc7a" path="/var/lib/kubelet/pods/9f7dae77-7a71-42b7-b625-903bdae5dc7a/volumes" Sep 07 10:40:19.572475 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.572439 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr"] Sep 07 10:40:19.577348 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.577332 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:19.579725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.579707 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:40:19.579832 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.579734 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:40:19.583393 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.583365 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr"] Sep 07 10:40:19.650860 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.650821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr\" (UID: \"e3455ddc-3f21-4a1f-b50f-b296145e38da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:19.751433 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.751407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr\" (UID: \"e3455ddc-3f21-4a1f-b50f-b296145e38da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:19.752009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.751991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr\" (UID: \"e3455ddc-3f21-4a1f-b50f-b296145e38da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:19.891401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:19.891325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:20.161106 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:20.161082 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr"] Sep 07 10:40:20.567555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:20.567502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" event={"ID":"e3455ddc-3f21-4a1f-b50f-b296145e38da","Type":"ContainerStarted","Data":"a19c3174479f5975257756b855501c9bc8d8881d6c9c8c822eebd74ccc33a9d0"} Sep 07 10:40:20.567780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:20.567568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" event={"ID":"e3455ddc-3f21-4a1f-b50f-b296145e38da","Type":"ContainerStarted","Data":"34e69b3d1a68e57a0fb075607d569d8829fb0820be974cfaf29ae7362f108cb6"} Sep 07 10:40:20.582464 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:20.582415 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" podStartSLOduration=1.582398594 podStartE2EDuration="1.582398594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:20.580233535 +0000 UTC m=+1188.437922851" watchObservedRunningTime="2026-09-07 10:40:20.582398594 +0000 UTC m=+1188.440087908" Sep 07 10:40:21.574543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:21.574509 2577 generic.go:356] "Generic (PLEG): container finished" podID="e3455ddc-3f21-4a1f-b50f-b296145e38da" containerID="a19c3174479f5975257756b855501c9bc8d8881d6c9c8c822eebd74ccc33a9d0" exitCode=0 Sep 07 10:40:21.574735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:21.574618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" event={"ID":"e3455ddc-3f21-4a1f-b50f-b296145e38da","Type":"ContainerDied","Data":"a19c3174479f5975257756b855501c9bc8d8881d6c9c8c822eebd74ccc33a9d0"} Sep 07 10:40:22.716005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:22.715982 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:22.779774 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:22.779744 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca\") pod \"e3455ddc-3f21-4a1f-b50f-b296145e38da\" (UID: \"e3455ddc-3f21-4a1f-b50f-b296145e38da\") " Sep 07 10:40:22.780072 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:22.780046 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3455ddc-3f21-4a1f-b50f-b296145e38da" (UID: "e3455ddc-3f21-4a1f-b50f-b296145e38da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:22.880973 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:22.880913 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3455ddc-3f21-4a1f-b50f-b296145e38da-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:23.587007 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:23.586980 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" Sep 07 10:40:23.587174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:23.586973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr" event={"ID":"e3455ddc-3f21-4a1f-b50f-b296145e38da","Type":"ContainerDied","Data":"34e69b3d1a68e57a0fb075607d569d8829fb0820be974cfaf29ae7362f108cb6"} Sep 07 10:40:23.587174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:23.587110 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34e69b3d1a68e57a0fb075607d569d8829fb0820be974cfaf29ae7362f108cb6" Sep 07 10:40:25.538389 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:25.538358 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:30.456736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:30.456695 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr"] Sep 07 10:40:30.459014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:30.458992 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-c5b29a-hsgzr"] Sep 07 10:40:30.728618 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:30.728558 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3455ddc-3f21-4a1f-b50f-b296145e38da" path="/var/lib/kubelet/pods/e3455ddc-3f21-4a1f-b50f-b296145e38da/volumes" Sep 07 10:40:35.665858 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.665826 2577 generic.go:356] "Generic (PLEG): container finished" podID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerID="4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c" exitCode=0 Sep 07 10:40:35.666396 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.665898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerDied","Data":"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c"} Sep 07 10:40:35.666396 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.666311 2577 scope.go:117] "RemoveContainer" containerID="4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c" Sep 07 10:40:35.667513 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.667474 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:35.667609 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.667571 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:35.667654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:35.667618 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:36.675303 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:36.675255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerStarted","Data":"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861"} Sep 07 10:40:36.828874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:36.828627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:36.830166 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:36.830114 2577 patch_prober.go:28] interesting pod/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:40:36.830312 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:36.830203 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 07 10:40:37.092227 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.092187 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8"] Sep 07 10:40:37.092679 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.092665 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3455ddc-3f21-4a1f-b50f-b296145e38da" containerName="revoke-keys" Sep 07 10:40:37.092731 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.092680 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3455ddc-3f21-4a1f-b50f-b296145e38da" containerName="revoke-keys" Sep 07 10:40:37.092767 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.092760 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3455ddc-3f21-4a1f-b50f-b296145e38da" containerName="revoke-keys" Sep 07 10:40:37.095727 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.095711 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:37.097546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.097522 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:40:37.097740 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.097523 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:40:37.102656 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.102632 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8"] Sep 07 10:40:37.133121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.131485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8\" (UID: \"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:37.232903 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.232867 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8\" (UID: \"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:37.233514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.233490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8\" (UID: \"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:37.413518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.413191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:37.697499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.697472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8"] Sep 07 10:40:37.699614 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:37.699556 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad51a9d_ce9e_45f5_9666_63bcbebcfe90.slice/crio-3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f WatchSource:0}: Error finding container 3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f: Status 404 returned error can't find the container with id 3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f Sep 07 10:40:37.829660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.829162 2577 patch_prober.go:28] interesting pod/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:40:37.829660 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:37.829229 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 07 10:40:38.687472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:38.687435 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" event={"ID":"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90","Type":"ContainerStarted","Data":"4db68f8dc5672238e6acfbcdf8839126b18eb65b1b79341c1966de01619f26d5"} Sep 07 10:40:38.687669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:38.687487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" event={"ID":"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90","Type":"ContainerStarted","Data":"3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f"} Sep 07 10:40:38.705000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:38.704955 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" podStartSLOduration=1.704936131 podStartE2EDuration="1.704936131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:38.70353286 +0000 UTC m=+1206.561222176" watchObservedRunningTime="2026-09-07 10:40:38.704936131 +0000 UTC m=+1206.562625449" Sep 07 10:40:38.828875 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:38.828830 2577 patch_prober.go:28] interesting pod/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:40:38.829026 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:38.828900 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 07 10:40:39.694538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:39.694499 2577 generic.go:356] "Generic (PLEG): container finished" podID="0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" containerID="4db68f8dc5672238e6acfbcdf8839126b18eb65b1b79341c1966de01619f26d5" exitCode=0 Sep 07 10:40:39.694784 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:39.694610 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" event={"ID":"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90","Type":"ContainerDied","Data":"4db68f8dc5672238e6acfbcdf8839126b18eb65b1b79341c1966de01619f26d5"} Sep 07 10:40:39.829032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:39.828993 2577 patch_prober.go:28] interesting pod/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:40:39.829613 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:39.829551 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 07 10:40:40.836700 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:40.835989 2577 patch_prober.go:28] interesting pod/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 07 10:40:40.836700 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:40.836052 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 07 10:40:40.871673 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:40.871646 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:40.968923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:40.968896 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca\") pod \"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90\" (UID: \"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90\") " Sep 07 10:40:40.969239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:40.969217 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" (UID: "0ad51a9d-ce9e-45f5-9666-63bcbebcfe90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:41.070070 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.070038 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:41.499663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.499627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:41.708760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.708732 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" Sep 07 10:40:41.708949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.708731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8" event={"ID":"0ad51a9d-ce9e-45f5-9666-63bcbebcfe90","Type":"ContainerDied","Data":"3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f"} Sep 07 10:40:41.708949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.708848 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cb2ee32fd2dcd8ec12ca0d01b998c1bf8bd4228692f2c2e66479724ef0cb21f" Sep 07 10:40:41.838324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.838236 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:41.839499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:41.839465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:42.811695 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:42.811659 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:42.811917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:42.811887 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" podUID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" containerName="payload-processing" containerID="cri-o://5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e" gracePeriod=30 Sep 07 10:40:42.999798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:42.999726 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8"] Sep 07 10:40:43.003192 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.003094 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc9185fe-l4bg8"] Sep 07 10:40:43.073380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.073323 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:43.193636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.193568 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwv7d\" (UniqueName: \"kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d\") pod \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " Sep 07 10:40:43.193811 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.193650 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume\") pod \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\" (UID: \"f65eb903-702a-49e8-a7bf-c2b2d80d6da2\") " Sep 07 10:40:43.194096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.194059 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f65eb903-702a-49e8-a7bf-c2b2d80d6da2" (UID: "f65eb903-702a-49e8-a7bf-c2b2d80d6da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:43.195894 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.195862 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d" (OuterVolumeSpecName: "kube-api-access-rwv7d") pod "f65eb903-702a-49e8-a7bf-c2b2d80d6da2" (UID: "f65eb903-702a-49e8-a7bf-c2b2d80d6da2"). InnerVolumeSpecName "kube-api-access-rwv7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:43.294941 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.294902 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwv7d\" (UniqueName: \"kubernetes.io/projected/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-kube-api-access-rwv7d\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:43.294941 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.294938 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65eb903-702a-49e8-a7bf-c2b2d80d6da2-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:43.721893 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.721855 2577 generic.go:356] "Generic (PLEG): container finished" podID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" containerID="5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e" exitCode=0 Sep 07 10:40:43.722086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.721915 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" Sep 07 10:40:43.722086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.721944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" event={"ID":"f65eb903-702a-49e8-a7bf-c2b2d80d6da2","Type":"ContainerDied","Data":"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e"} Sep 07 10:40:43.722086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.721992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8" event={"ID":"f65eb903-702a-49e8-a7bf-c2b2d80d6da2","Type":"ContainerDied","Data":"92711be8dab398b337d36eb98cbd1fcf5cdb91f41d3f287831cb5b387600eef6"} Sep 07 10:40:43.722086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.722016 2577 scope.go:117] "RemoveContainer" containerID="5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e" Sep 07 10:40:43.733842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.733819 2577 scope.go:117] "RemoveContainer" containerID="5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e" Sep 07 10:40:43.734094 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:40:43.734074 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e\": container with ID starting with 5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e not found: ID does not exist" containerID="5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e" Sep 07 10:40:43.734164 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.734101 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e"} err="failed to get container status \"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e\": rpc error: code = NotFound desc = could not find container \"5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e\": container with ID starting with 5646fa11057133c756e2160d06bb8e180c14dfbd904b6260044b9a8212691c6e not found: ID does not exist" Sep 07 10:40:43.746356 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.746336 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:43.749637 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:43.749616 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9185fe-89ff4db9-l5gn8"] Sep 07 10:40:44.470504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:44.470467 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:44.471053 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:44.470747 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" containerID="cri-o://cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861" gracePeriod=30 Sep 07 10:40:44.723067 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:44.722973 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" path="/var/lib/kubelet/pods/0ad51a9d-ce9e-45f5-9666-63bcbebcfe90/volumes" Sep 07 10:40:44.723422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:44.723407 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" path="/var/lib/kubelet/pods/f65eb903-702a-49e8-a7bf-c2b2d80d6da2/volumes" Sep 07 10:40:45.355978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.355948 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:40:45.356439 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356423 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" containerName="payload-processing" Sep 07 10:40:45.356509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356441 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" containerName="payload-processing" Sep 07 10:40:45.356509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" containerName="revoke-keys" Sep 07 10:40:45.356509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356461 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" containerName="revoke-keys" Sep 07 10:40:45.356697 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356548 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad51a9d-ce9e-45f5-9666-63bcbebcfe90" containerName="revoke-keys" Sep 07 10:40:45.356697 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.356561 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65eb903-702a-49e8-a7bf-c2b2d80d6da2" containerName="payload-processing" Sep 07 10:40:45.361111 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.361091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.363194 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.363173 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1ccb3bcb-openshift-default-dockercfg-cfsct\"" Sep 07 10:40:45.372816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.372791 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:40:45.415050 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8b4\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.415434 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.415370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516402 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516473 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516577 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516833 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.516895 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516880 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.517341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.516926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8b4\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.517341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.517133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.517341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.517194 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.517443 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.517407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.519112 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.519091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.519266 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.519151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.524075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.524052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.524196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.524121 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8b4\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4\") pod \"e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.674345 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.674252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:45.954235 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.954211 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:40:45.955778 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:45.955755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4955b4_cf9b_4866_ab18_ec16b01fff79.slice/crio-a90581699da88a44abb6a92d2ce298b60c9494db0ead0684e466c628c838cd6c WatchSource:0}: Error finding container a90581699da88a44abb6a92d2ce298b60c9494db0ead0684e466c628c838cd6c: Status 404 returned error can't find the container with id a90581699da88a44abb6a92d2ce298b60c9494db0ead0684e466c628c838cd6c Sep 07 10:40:45.958474 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.958446 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:45.958556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.958514 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:45.958556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:45.958544 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:40:46.744268 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:46.744227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerStarted","Data":"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b"} Sep 07 10:40:46.744776 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:46.744290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerStarted","Data":"a90581699da88a44abb6a92d2ce298b60c9494db0ead0684e466c628c838cd6c"} Sep 07 10:40:46.765507 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:46.765450 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podStartSLOduration=1.765436241 podStartE2EDuration="1.765436241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:46.76199789 +0000 UTC m=+1214.619687232" watchObservedRunningTime="2026-09-07 10:40:46.765436241 +0000 UTC m=+1214.623125556" Sep 07 10:40:47.674735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:47.674701 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:47.679829 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:47.679806 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:47.750329 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:47.750299 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:47.751017 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:47.751000 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:40:49.725511 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.725486 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:49.762156 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.762078 2577 generic.go:356] "Generic (PLEG): container finished" podID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerID="cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861" exitCode=0 Sep 07 10:40:49.762156 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.762127 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" Sep 07 10:40:49.762334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.762164 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerDied","Data":"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861"} Sep 07 10:40:49.762334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.762221 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv" event={"ID":"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2","Type":"ContainerDied","Data":"df081ef41eb84ca3fa6085b22822b54555b1042d73442d2cf81870701a602fb8"} Sep 07 10:40:49.762334 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.762254 2577 scope.go:117] "RemoveContainer" containerID="cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861" Sep 07 10:40:49.771760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.771745 2577 scope.go:117] "RemoveContainer" containerID="4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c" Sep 07 10:40:49.780531 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.780509 2577 scope.go:117] "RemoveContainer" containerID="cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861" Sep 07 10:40:49.780885 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:40:49.780866 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861\": container with ID starting with cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861 not found: ID does not exist" containerID="cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861" Sep 07 10:40:49.780954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.780901 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861"} err="failed to get container status \"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861\": rpc error: code = NotFound desc = could not find container \"cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861\": container with ID starting with cd6db4b98468f460dfd4b1e0cf43002f5689ffb26e51fd7eeb8d84117cbac861 not found: ID does not exist" Sep 07 10:40:49.780954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.780918 2577 scope.go:117] "RemoveContainer" containerID="4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c" Sep 07 10:40:49.781188 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:40:49.781169 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c\": container with ID starting with 4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c not found: ID does not exist" containerID="4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c" Sep 07 10:40:49.781243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.781199 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c"} err="failed to get container status \"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c\": rpc error: code = NotFound desc = could not find container \"4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c\": container with ID starting with 4de20f0dc345ed8c7b89a981ad8e32de814c6d97866d9de2b48a60c1a44e904c not found: ID does not exist" Sep 07 10:40:49.864469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864444 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864482 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864529 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864614 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864637 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhzgv\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.864848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864822 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy\") pod \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\" (UID: \"3b11de89-deba-4c46-ae2a-a48f8d8b4ee2\") " Sep 07 10:40:49.865047 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:49.865047 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.864997 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:49.865154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.865050 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:40:49.865715 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.865644 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:49.866198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.866140 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.866198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.866162 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.866198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.866174 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.866198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.866187 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.866403 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.866258 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:49.867138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.867116 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv" (OuterVolumeSpecName: "kube-api-access-vhzgv") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "kube-api-access-vhzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:49.867223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.867194 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:40:49.867467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.867447 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:40:49.867530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.867505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" (UID: "3b11de89-deba-4c46-ae2a-a48f8d8b4ee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:40:49.967411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.967384 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.967411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.967409 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.967627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.967419 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzgv\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-kube-api-access-vhzgv\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.967627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.967428 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:49.967627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:49.967436 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:40:50.086445 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.086416 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:50.088461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.088439 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fc9185fe-openshift-default-7d6bf9b65d-cscbv"] Sep 07 10:40:50.123754 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.123733 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:40:50.124180 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.124168 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.124229 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.124182 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.124229 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.124196 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.124229 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.124201 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.124362 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.124278 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.128848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.128833 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.131092 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.131069 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1ccb3bcb-dockercfg-6wpll\"" Sep 07 10:40:50.131167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.131094 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1ccb3bcb\"" Sep 07 10:40:50.136890 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.136870 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:40:50.180290 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.180264 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:40:50.180845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.180829 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" containerName="istio-proxy" Sep 07 10:40:50.183869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.183855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.194048 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.194023 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:40:50.270147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.270098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9z6t\" (UniqueName: \"kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.270336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.270307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.270409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.270339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbz5\" (UniqueName: \"kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.270409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.270363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.371896 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.371818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9z6t\" (UniqueName: \"kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.372045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.371926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.372045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.371946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbz5\" (UniqueName: \"kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.372045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.371969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.372637 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.372619 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.372702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.372675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.379631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.379602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbz5\" (UniqueName: \"kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5\") pod \"payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.379998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.379978 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9z6t\" (UniqueName: \"kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t\") pod \"payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.440896 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.440866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:50.494771 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.494748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:50.713023 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.712994 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:40:50.714833 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:50.714805 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af65738_41fd_4edf_92d8_2d372cd2bcf6.slice/crio-f8ad7508e453c325bd2b632d73adf815f2d8889936ca87cbc361d0b1917f7db6 WatchSource:0}: Error finding container f8ad7508e453c325bd2b632d73adf815f2d8889936ca87cbc361d0b1917f7db6: Status 404 returned error can't find the container with id f8ad7508e453c325bd2b632d73adf815f2d8889936ca87cbc361d0b1917f7db6 Sep 07 10:40:50.735104 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.735075 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b11de89-deba-4c46-ae2a-a48f8d8b4ee2" path="/var/lib/kubelet/pods/3b11de89-deba-4c46-ae2a-a48f8d8b4ee2/volumes" Sep 07 10:40:50.768832 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.768800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" event={"ID":"9af65738-41fd-4edf-92d8-2d372cd2bcf6","Type":"ContainerStarted","Data":"f8ad7508e453c325bd2b632d73adf815f2d8889936ca87cbc361d0b1917f7db6"} Sep 07 10:40:50.784930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:50.784907 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:40:50.789871 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:40:50.789848 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70b936e_4d56_4d75_8b4f_07516b0b048d.slice/crio-a2bdc1efc44e02dfd873d61eb7b51f97fb943951467eb2477eb5b72f6ac5c4a0 WatchSource:0}: Error finding container a2bdc1efc44e02dfd873d61eb7b51f97fb943951467eb2477eb5b72f6ac5c4a0: Status 404 returned error can't find the container with id a2bdc1efc44e02dfd873d61eb7b51f97fb943951467eb2477eb5b72f6ac5c4a0 Sep 07 10:40:51.777541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.777503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" event={"ID":"9af65738-41fd-4edf-92d8-2d372cd2bcf6","Type":"ContainerStarted","Data":"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2"} Sep 07 10:40:51.778005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.777601 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:40:51.779185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.779159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" event={"ID":"a70b936e-4d56-4d75-8b4f-07516b0b048d","Type":"ContainerStarted","Data":"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a"} Sep 07 10:40:51.779318 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.779200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" event={"ID":"a70b936e-4d56-4d75-8b4f-07516b0b048d","Type":"ContainerStarted","Data":"a2bdc1efc44e02dfd873d61eb7b51f97fb943951467eb2477eb5b72f6ac5c4a0"} Sep 07 10:40:51.779318 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.779311 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:40:51.794954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.794914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" podStartSLOduration=1.7949018589999999 podStartE2EDuration="1.794901859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:51.792503952 +0000 UTC m=+1219.650193269" watchObservedRunningTime="2026-09-07 10:40:51.794901859 +0000 UTC m=+1219.652591176" Sep 07 10:40:51.807303 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:40:51.807262 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" podStartSLOduration=1.807249696 podStartE2EDuration="1.807249696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:40:51.806690679 +0000 UTC m=+1219.664379998" watchObservedRunningTime="2026-09-07 10:40:51.807249696 +0000 UTC m=+1219.664939012" Sep 07 10:41:02.785653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:02.785617 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:41:02.786121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:02.785687 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:41:03.792716 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:03.792682 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:41:03.793065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:03.792748 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:41:04.864877 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.864843 2577 generic.go:356] "Generic (PLEG): container finished" podID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerID="63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b" exitCode=0 Sep 07 10:41:04.865279 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.864922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerDied","Data":"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b"} Sep 07 10:41:04.865379 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.865365 2577 scope.go:117] "RemoveContainer" containerID="63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b" Sep 07 10:41:04.866548 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.866521 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:41:04.866631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.866618 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:41:04.866678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:04.866657 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:41:05.871735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:05.871699 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerStarted","Data":"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834"} Sep 07 10:41:06.675381 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:06.675345 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:06.675562 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:06.675542 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 07 10:41:06.675657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:06.675635 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 07 10:41:07.675257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:07.675223 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 07 10:41:07.675693 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:07.675275 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 07 10:41:08.675230 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:08.675194 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 07 10:41:08.675411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:08.675260 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 07 10:41:09.674949 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:09.674915 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 07 10:41:09.675128 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:09.674969 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 07 10:41:10.236620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.228956 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:41:10.236620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.229202 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" containerID="cri-o://e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4" gracePeriod=30 Sep 07 10:41:10.674890 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.674796 2577 patch_prober.go:28] interesting pod/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Sep 07 10:41:10.674890 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.674869 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Sep 07 10:41:10.803308 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.803271 2577 patch_prober.go:28] interesting pod/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:41:10.803476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:10.803331 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:41:11.678813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:11.678779 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:11.679189 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:11.679086 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:11.679757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:11.679741 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:15.493209 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.493181 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:41:15.515359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515267 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515326 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqjdr\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515379 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515412 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515449 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515494 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515615 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.515657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515656 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.516118 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.515678 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy\") pod \"ddaed435-bb00-42b9-979e-847262f2b06b\" (UID: \"ddaed435-bb00-42b9-979e-847262f2b06b\") " Sep 07 10:41:15.516554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.516527 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:15.516702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.516544 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:15.516702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.516638 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:15.516813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.516727 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data" (OuterVolumeSpecName: "istio-data") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:15.516867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.516831 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:15.518812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.518752 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:15.519159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.519128 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token" (OuterVolumeSpecName: "istio-token") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:15.519159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.519140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr" (OuterVolumeSpecName: "kube-api-access-nqjdr") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "kube-api-access-nqjdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:15.519325 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.519175 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ddaed435-bb00-42b9-979e-847262f2b06b" (UID: "ddaed435-bb00-42b9-979e-847262f2b06b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:15.616594 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616559 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616614 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddaed435-bb00-42b9-979e-847262f2b06b-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616623 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616631 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616639 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616646 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616656 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddaed435-bb00-42b9-979e-847262f2b06b-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616664 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddaed435-bb00-42b9-979e-847262f2b06b-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.616755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.616671 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjdr\" (UniqueName: \"kubernetes.io/projected/ddaed435-bb00-42b9-979e-847262f2b06b-kube-api-access-nqjdr\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:15.934612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.934515 2577 generic.go:356] "Generic (PLEG): container finished" podID="ddaed435-bb00-42b9-979e-847262f2b06b" containerID="e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4" exitCode=0 Sep 07 10:41:15.934612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.934565 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" Sep 07 10:41:15.934787 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.934623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerDied","Data":"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4"} Sep 07 10:41:15.934787 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.934681 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh" event={"ID":"ddaed435-bb00-42b9-979e-847262f2b06b","Type":"ContainerDied","Data":"68b0f8e0dfd05b56d30d082f2c570ec0fb2a39a031729cf60fcf403aac19d62d"} Sep 07 10:41:15.934787 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.934704 2577 scope.go:117] "RemoveContainer" containerID="e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4" Sep 07 10:41:15.944355 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.944338 2577 scope.go:117] "RemoveContainer" containerID="334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e" Sep 07 10:41:15.953252 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.953239 2577 scope.go:117] "RemoveContainer" containerID="e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4" Sep 07 10:41:15.953526 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:15.953508 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4\": container with ID starting with e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4 not found: ID does not exist" containerID="e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4" Sep 07 10:41:15.953577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.953534 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4"} err="failed to get container status \"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4\": rpc error: code = NotFound desc = could not find container \"e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4\": container with ID starting with e64405c4919b69ded012b1d5d3507453b6855e2dc4fe453a9d24c836be92c9a4 not found: ID does not exist" Sep 07 10:41:15.953577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.953552 2577 scope.go:117] "RemoveContainer" containerID="334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e" Sep 07 10:41:15.953822 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:15.953803 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e\": container with ID starting with 334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e not found: ID does not exist" containerID="334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e" Sep 07 10:41:15.953874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.953833 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e"} err="failed to get container status \"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e\": rpc error: code = NotFound desc = could not find container \"334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e\": container with ID starting with 334d489448b71c0e827e06ac099321d49c69f07c91083776eff944a01ffe1b0e not found: ID does not exist" Sep 07 10:41:15.958980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.958768 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:41:15.961797 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:15.961780 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c5b29a-openshift-default-c4bf8b84b-f6nmh"] Sep 07 10:41:16.057365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057329 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:41:16.057804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057790 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:16.057804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057804 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:16.057910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057819 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:16.057910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057827 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:16.057971 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.057914 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:16.060617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.060574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.062768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.062748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a49ef7-dockercfg-7g7ck\"" Sep 07 10:41:16.062873 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.062774 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a49ef7\"" Sep 07 10:41:16.068741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.068719 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:41:16.120948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.120924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.121065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.120954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lcq\" (UniqueName: \"kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.221874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.221845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.222029 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.221883 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lcq\" (UniqueName: \"kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.222755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.222735 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.230311 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.230284 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lcq\" (UniqueName: \"kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq\") pod \"payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.372330 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.372290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.650083 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.650004 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:41:16.653512 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:41:16.653481 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759a1243_4e18_4fdd_bec9_a66d21535290.slice/crio-1cc344cc9b4a682ef7a8ba6f388041f2da64a0da39a6969654c8d02048b7a085 WatchSource:0}: Error finding container 1cc344cc9b4a682ef7a8ba6f388041f2da64a0da39a6969654c8d02048b7a085: Status 404 returned error can't find the container with id 1cc344cc9b4a682ef7a8ba6f388041f2da64a0da39a6969654c8d02048b7a085 Sep 07 10:41:16.725213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.725187 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" path="/var/lib/kubelet/pods/ddaed435-bb00-42b9-979e-847262f2b06b/volumes" Sep 07 10:41:16.943358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.943273 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" event={"ID":"759a1243-4e18-4fdd-bec9-a66d21535290","Type":"ContainerStarted","Data":"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163"} Sep 07 10:41:16.943358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.943331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" event={"ID":"759a1243-4e18-4fdd-bec9-a66d21535290","Type":"ContainerStarted","Data":"1cc344cc9b4a682ef7a8ba6f388041f2da64a0da39a6969654c8d02048b7a085"} Sep 07 10:41:16.943555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.943414 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:16.964717 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:16.964675 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" podStartSLOduration=0.964659172 podStartE2EDuration="964.659172ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:41:16.963919771 +0000 UTC m=+1244.821609085" watchObservedRunningTime="2026-09-07 10:41:16.964659172 +0000 UTC m=+1244.822348487" Sep 07 10:41:20.216232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.216197 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:41:20.216711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.216500 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" podUID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" containerName="payload-processing" containerID="cri-o://88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2" gracePeriod=30 Sep 07 10:41:20.225599 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.225558 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:41:20.226254 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.226181 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" podUID="a70b936e-4d56-4d75-8b4f-07516b0b048d" containerName="payload-pre-processing" containerID="cri-o://ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a" gracePeriod=30 Sep 07 10:41:20.496136 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.496106 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:41:20.530525 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.530498 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:41:20.539619 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.539566 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:41:20.539847 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.539815 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" containerID="cri-o://2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834" gracePeriod=30 Sep 07 10:41:20.570972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.570940 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9z6t\" (UniqueName: \"kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t\") pod \"a70b936e-4d56-4d75-8b4f-07516b0b048d\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " Sep 07 10:41:20.571112 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.571037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume\") pod \"a70b936e-4d56-4d75-8b4f-07516b0b048d\" (UID: \"a70b936e-4d56-4d75-8b4f-07516b0b048d\") " Sep 07 10:41:20.571185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.571115 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bbz5\" (UniqueName: \"kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5\") pod \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " Sep 07 10:41:20.571185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.571154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume\") pod \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\" (UID: \"9af65738-41fd-4edf-92d8-2d372cd2bcf6\") " Sep 07 10:41:20.571516 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.571471 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a70b936e-4d56-4d75-8b4f-07516b0b048d" (UID: "a70b936e-4d56-4d75-8b4f-07516b0b048d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:20.571651 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.571576 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9af65738-41fd-4edf-92d8-2d372cd2bcf6" (UID: "9af65738-41fd-4edf-92d8-2d372cd2bcf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:20.573750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.573723 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t" (OuterVolumeSpecName: "kube-api-access-k9z6t") pod "a70b936e-4d56-4d75-8b4f-07516b0b048d" (UID: "a70b936e-4d56-4d75-8b4f-07516b0b048d"). InnerVolumeSpecName "kube-api-access-k9z6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:20.573839 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.573761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5" (OuterVolumeSpecName: "kube-api-access-4bbz5") pod "9af65738-41fd-4edf-92d8-2d372cd2bcf6" (UID: "9af65738-41fd-4edf-92d8-2d372cd2bcf6"). InnerVolumeSpecName "kube-api-access-4bbz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:20.672314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.672277 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbz5\" (UniqueName: \"kubernetes.io/projected/9af65738-41fd-4edf-92d8-2d372cd2bcf6-kube-api-access-4bbz5\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:20.672314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.672307 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af65738-41fd-4edf-92d8-2d372cd2bcf6-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:20.672314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.672316 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9z6t\" (UniqueName: \"kubernetes.io/projected/a70b936e-4d56-4d75-8b4f-07516b0b048d-kube-api-access-k9z6t\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:20.672543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.672326 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a70b936e-4d56-4d75-8b4f-07516b0b048d-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:20.966720 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.966681 2577 generic.go:356] "Generic (PLEG): container finished" podID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" containerID="88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2" exitCode=0 Sep 07 10:41:20.966720 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.966722 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" Sep 07 10:41:20.966970 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.966766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" event={"ID":"9af65738-41fd-4edf-92d8-2d372cd2bcf6","Type":"ContainerDied","Data":"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2"} Sep 07 10:41:20.966970 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.966821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82" event={"ID":"9af65738-41fd-4edf-92d8-2d372cd2bcf6","Type":"ContainerDied","Data":"f8ad7508e453c325bd2b632d73adf815f2d8889936ca87cbc361d0b1917f7db6"} Sep 07 10:41:20.966970 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.966845 2577 scope.go:117] "RemoveContainer" containerID="88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2" Sep 07 10:41:20.968276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.968252 2577 generic.go:356] "Generic (PLEG): container finished" podID="a70b936e-4d56-4d75-8b4f-07516b0b048d" containerID="ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a" exitCode=0 Sep 07 10:41:20.968414 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.968297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" event={"ID":"a70b936e-4d56-4d75-8b4f-07516b0b048d","Type":"ContainerDied","Data":"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a"} Sep 07 10:41:20.968414 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.968307 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" Sep 07 10:41:20.968414 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.968327 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz" event={"ID":"a70b936e-4d56-4d75-8b4f-07516b0b048d","Type":"ContainerDied","Data":"a2bdc1efc44e02dfd873d61eb7b51f97fb943951467eb2477eb5b72f6ac5c4a0"} Sep 07 10:41:20.976625 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.976609 2577 scope.go:117] "RemoveContainer" containerID="88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2" Sep 07 10:41:20.976885 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:20.976868 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2\": container with ID starting with 88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2 not found: ID does not exist" containerID="88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2" Sep 07 10:41:20.976931 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.976894 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2"} err="failed to get container status \"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2\": rpc error: code = NotFound desc = could not find container \"88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2\": container with ID starting with 88bc83aaffaf16b7a007a9d65ada3fe9b48d586f9d4311de69b534b4f47be6a2 not found: ID does not exist" Sep 07 10:41:20.976931 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.976909 2577 scope.go:117] "RemoveContainer" containerID="ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a" Sep 07 10:41:20.985131 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.985113 2577 scope.go:117] "RemoveContainer" containerID="ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a" Sep 07 10:41:20.985422 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:20.985405 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a\": container with ID starting with ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a not found: ID does not exist" containerID="ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a" Sep 07 10:41:20.985476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.985431 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a"} err="failed to get container status \"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a\": rpc error: code = NotFound desc = could not find container \"ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a\": container with ID starting with ca255a913d57a29f5c5655191a6a68886ff6e3c466231f8f8a9a3f6119b6d26a not found: ID does not exist" Sep 07 10:41:20.988539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.988519 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:41:20.992263 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:20.992238 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1ccb3bcb-54bc5c75f7-znb82"] Sep 07 10:41:21.001539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:21.001521 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:41:21.005552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:21.005533 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1ccb3bcb-678c86d774-7gzgz"] Sep 07 10:41:22.723520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:22.723489 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" path="/var/lib/kubelet/pods/9af65738-41fd-4edf-92d8-2d372cd2bcf6/volumes" Sep 07 10:41:22.723886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:22.723863 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70b936e-4d56-4d75-8b4f-07516b0b048d" path="/var/lib/kubelet/pods/a70b936e-4d56-4d75-8b4f-07516b0b048d/volumes" Sep 07 10:41:25.804218 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.804194 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:25.919365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919321 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919339 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t8b4\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919355 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919462 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919492 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919517 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919540 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy\") pod \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\" (UID: \"3e4955b4-cf9b-4866-ab18-ec16b01fff79\") " Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:25.919710 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919727 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919746 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919971 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919990 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.919995 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:41:25.920028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.920007 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:25.920265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.920038 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:25.921463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.921442 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4" (OuterVolumeSpecName: "kube-api-access-6t8b4") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "kube-api-access-6t8b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:25.921886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.921871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:41:25.922258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.922232 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:41:25.922258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.922244 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e4955b4-cf9b-4866-ab18-ec16b01fff79" (UID: "3e4955b4-cf9b-4866-ab18-ec16b01fff79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:41:25.999862 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.999834 2577 generic.go:356] "Generic (PLEG): container finished" podID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerID="2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834" exitCode=0 Sep 07 10:41:25.999956 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.999880 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" Sep 07 10:41:25.999956 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.999875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerDied","Data":"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834"} Sep 07 10:41:26.000028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:25.999990 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m" event={"ID":"3e4955b4-cf9b-4866-ab18-ec16b01fff79","Type":"ContainerDied","Data":"a90581699da88a44abb6a92d2ce298b60c9494db0ead0684e466c628c838cd6c"} Sep 07 10:41:26.000028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.000014 2577 scope.go:117] "RemoveContainer" containerID="2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834" Sep 07 10:41:26.009267 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.009251 2577 scope.go:117] "RemoveContainer" containerID="63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b" Sep 07 10:41:26.017224 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.017209 2577 scope.go:117] "RemoveContainer" containerID="2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834" Sep 07 10:41:26.017652 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:26.017563 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834\": container with ID starting with 2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834 not found: ID does not exist" containerID="2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834" Sep 07 10:41:26.017652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.017632 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834"} err="failed to get container status \"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834\": rpc error: code = NotFound desc = could not find container \"2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834\": container with ID starting with 2b3ecca75a7802588b398eddb31e7bf4773153c9fde75bb1ca80885bb4153834 not found: ID does not exist" Sep 07 10:41:26.017830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.017661 2577 scope.go:117] "RemoveContainer" containerID="63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b" Sep 07 10:41:26.018023 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:41:26.017957 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b\": container with ID starting with 63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b not found: ID does not exist" containerID="63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b" Sep 07 10:41:26.018023 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.017986 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b"} err="failed to get container status \"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b\": rpc error: code = NotFound desc = could not find container \"63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b\": container with ID starting with 63a328b9dcf76ed62eec6bd221801b9226047064fc8e7c6bb0cc645d214ff59b not found: ID does not exist" Sep 07 10:41:26.020664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020645 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:41:26.020747 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020728 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:26.020747 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020744 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:26.020822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020752 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:26.020822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020759 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e4955b4-cf9b-4866-ab18-ec16b01fff79-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:26.020822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.020766 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8b4\" (UniqueName: \"kubernetes.io/projected/3e4955b4-cf9b-4866-ab18-ec16b01fff79-kube-api-access-6t8b4\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:41:26.025052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.025034 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1ccb3bcb-openshift-default-7689fb8b9b-wtv2m"] Sep 07 10:41:26.723981 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:26.723952 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" path="/var/lib/kubelet/pods/3e4955b4-cf9b-4866-ab18-ec16b01fff79/volumes" Sep 07 10:41:27.951110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:27.951083 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:41:59.422711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.422671 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj"] Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423103 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423114 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423127 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70b936e-4d56-4d75-8b4f-07516b0b048d" containerName="payload-pre-processing" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423133 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70b936e-4d56-4d75-8b4f-07516b0b048d" containerName="payload-pre-processing" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423145 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" containerName="payload-processing" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423150 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" containerName="payload-processing" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423207 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423214 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70b936e-4d56-4d75-8b4f-07516b0b048d" containerName="payload-pre-processing" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423219 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddaed435-bb00-42b9-979e-847262f2b06b" containerName="istio-proxy" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423225 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:41:59.423288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.423234 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af65738-41fd-4edf-92d8-2d372cd2bcf6" containerName="payload-processing" Sep 07 10:41:59.428694 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.428666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:41:59.432674 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.432646 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:41:59.432807 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.432676 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:41:59.435400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.435378 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj"] Sep 07 10:41:59.526422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.526385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj\" (UID: \"e93a49fc-2a8f-490a-a7cb-5215c9024580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:41:59.627643 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.627614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj\" (UID: \"e93a49fc-2a8f-490a-a7cb-5215c9024580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:41:59.628188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.628168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj\" (UID: \"e93a49fc-2a8f-490a-a7cb-5215c9024580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:41:59.742113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:41:59.742090 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:42:00.023995 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:00.023906 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj"] Sep 07 10:42:00.029573 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:42:00.029530 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93a49fc_2a8f_490a_a7cb_5215c9024580.slice/crio-f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d WatchSource:0}: Error finding container f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d: Status 404 returned error can't find the container with id f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d Sep 07 10:42:00.190409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:00.190376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" event={"ID":"e93a49fc-2a8f-490a-a7cb-5215c9024580","Type":"ContainerStarted","Data":"f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d"} Sep 07 10:42:01.198366 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:01.198332 2577 generic.go:356] "Generic (PLEG): container finished" podID="e93a49fc-2a8f-490a-a7cb-5215c9024580" containerID="2f1f9d2cf7ff8fd731d7610b72e97ca6449d08cd72bc4d192ea3b8422a35add9" exitCode=0 Sep 07 10:42:01.198763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:01.198469 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" event={"ID":"e93a49fc-2a8f-490a-a7cb-5215c9024580","Type":"ContainerDied","Data":"2f1f9d2cf7ff8fd731d7610b72e97ca6449d08cd72bc4d192ea3b8422a35add9"} Sep 07 10:42:02.346013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:02.345989 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:42:02.454567 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:02.454539 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca\") pod \"e93a49fc-2a8f-490a-a7cb-5215c9024580\" (UID: \"e93a49fc-2a8f-490a-a7cb-5215c9024580\") " Sep 07 10:42:02.454924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:02.454872 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e93a49fc-2a8f-490a-a7cb-5215c9024580" (UID: "e93a49fc-2a8f-490a-a7cb-5215c9024580"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:02.555400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:02.555372 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e93a49fc-2a8f-490a-a7cb-5215c9024580-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:03.210518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:03.210436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" event={"ID":"e93a49fc-2a8f-490a-a7cb-5215c9024580","Type":"ContainerDied","Data":"f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d"} Sep 07 10:42:03.210518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:03.210457 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj" Sep 07 10:42:03.210518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:03.210491 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40e8066e274075067f679b162b7a1f226a9d8b3eb360636586f76e7ac57071d" Sep 07 10:42:09.953319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:09.953268 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:42:09.953840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:09.953543 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" podUID="759a1243-4e18-4fdd-bec9-a66d21535290" containerName="payload-pre-processing" containerID="cri-o://b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163" gracePeriod=30 Sep 07 10:42:10.125834 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.125795 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj"] Sep 07 10:42:10.128046 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.128021 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a49ef7-dpmqj"] Sep 07 10:42:10.210191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.210131 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:42:10.247653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.247627 2577 generic.go:356] "Generic (PLEG): container finished" podID="759a1243-4e18-4fdd-bec9-a66d21535290" containerID="b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163" exitCode=0 Sep 07 10:42:10.247806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.247665 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" Sep 07 10:42:10.247806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.247702 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" event={"ID":"759a1243-4e18-4fdd-bec9-a66d21535290","Type":"ContainerDied","Data":"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163"} Sep 07 10:42:10.247806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.247743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb" event={"ID":"759a1243-4e18-4fdd-bec9-a66d21535290","Type":"ContainerDied","Data":"1cc344cc9b4a682ef7a8ba6f388041f2da64a0da39a6969654c8d02048b7a085"} Sep 07 10:42:10.247806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.247766 2577 scope.go:117] "RemoveContainer" containerID="b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163" Sep 07 10:42:10.258386 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.258370 2577 scope.go:117] "RemoveContainer" containerID="b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163" Sep 07 10:42:10.258657 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:42:10.258637 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163\": container with ID starting with b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163 not found: ID does not exist" containerID="b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163" Sep 07 10:42:10.258760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.258662 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163"} err="failed to get container status \"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163\": rpc error: code = NotFound desc = could not find container \"b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163\": container with ID starting with b6863a019e2e750eac6c7866e47a0280effa8663ecc52608b9c0bef4eb1bb163 not found: ID does not exist" Sep 07 10:42:10.328708 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.328677 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9lcq\" (UniqueName: \"kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq\") pod \"759a1243-4e18-4fdd-bec9-a66d21535290\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " Sep 07 10:42:10.328874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.328737 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume\") pod \"759a1243-4e18-4fdd-bec9-a66d21535290\" (UID: \"759a1243-4e18-4fdd-bec9-a66d21535290\") " Sep 07 10:42:10.329135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.329105 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "759a1243-4e18-4fdd-bec9-a66d21535290" (UID: "759a1243-4e18-4fdd-bec9-a66d21535290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:10.331063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.331038 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq" (OuterVolumeSpecName: "kube-api-access-q9lcq") pod "759a1243-4e18-4fdd-bec9-a66d21535290" (UID: "759a1243-4e18-4fdd-bec9-a66d21535290"). InnerVolumeSpecName "kube-api-access-q9lcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:42:10.430074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.430040 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lcq\" (UniqueName: \"kubernetes.io/projected/759a1243-4e18-4fdd-bec9-a66d21535290-kube-api-access-q9lcq\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:10.430074 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.430070 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/759a1243-4e18-4fdd-bec9-a66d21535290-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:10.578600 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.578558 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:42:10.588075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.588044 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a49ef7-6896c6b6d6-t2jqb"] Sep 07 10:42:10.724419 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.724382 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="759a1243-4e18-4fdd-bec9-a66d21535290" path="/var/lib/kubelet/pods/759a1243-4e18-4fdd-bec9-a66d21535290/volumes" Sep 07 10:42:10.724943 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:10.724910 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e93a49fc-2a8f-490a-a7cb-5215c9024580" path="/var/lib/kubelet/pods/e93a49fc-2a8f-490a-a7cb-5215c9024580/volumes" Sep 07 10:42:17.479799 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.479768 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:17.480315 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e93a49fc-2a8f-490a-a7cb-5215c9024580" containerName="revoke-keys" Sep 07 10:42:17.480357 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480320 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e93a49fc-2a8f-490a-a7cb-5215c9024580" containerName="revoke-keys" Sep 07 10:42:17.480357 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480333 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="759a1243-4e18-4fdd-bec9-a66d21535290" containerName="payload-pre-processing" Sep 07 10:42:17.480357 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480342 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="759a1243-4e18-4fdd-bec9-a66d21535290" containerName="payload-pre-processing" Sep 07 10:42:17.480458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480390 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:42:17.480458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480398 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4955b4-cf9b-4866-ab18-ec16b01fff79" containerName="istio-proxy" Sep 07 10:42:17.480519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480499 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="759a1243-4e18-4fdd-bec9-a66d21535290" containerName="payload-pre-processing" Sep 07 10:42:17.480519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.480514 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e93a49fc-2a8f-490a-a7cb-5215c9024580" containerName="revoke-keys" Sep 07 10:42:17.487270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.484776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.488312 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.488286 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-2b5f38-openshift-default-dockercfg-q9rlx\"" Sep 07 10:42:17.497070 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.497042 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:17.596264 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596603 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9z4c\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.596719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.596642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697364 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9z4c\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697791 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697929 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.697987 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697958 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.698042 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.697977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.698080 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.698031 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.698129 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.698085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.698532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.698461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.700063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.700043 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.700139 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.700101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.706762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.706744 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.707004 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.706986 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9z4c\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c\") pod \"e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:17.801811 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:17.801757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:18.131530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.131507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:18.134272 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:42:18.134244 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe810b58_a843_4d04_ae70_5f73bc261329.slice/crio-218f8bf0452ce5f02bb7d60168a6a25bd4f3a0c42d071e4f2746315fd50cdb94 WatchSource:0}: Error finding container 218f8bf0452ce5f02bb7d60168a6a25bd4f3a0c42d071e4f2746315fd50cdb94: Status 404 returned error can't find the container with id 218f8bf0452ce5f02bb7d60168a6a25bd4f3a0c42d071e4f2746315fd50cdb94 Sep 07 10:42:18.136404 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.136377 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:18.136477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.136442 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:18.136477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.136470 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:18.296378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.296335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" event={"ID":"fe810b58-a843-4d04-ae70-5f73bc261329","Type":"ContainerStarted","Data":"3e22c28b23bc00b4b7a9e969e8868a06ffc362164373bfeb1725701dd039e961"} Sep 07 10:42:18.296571 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.296408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" event={"ID":"fe810b58-a843-4d04-ae70-5f73bc261329","Type":"ContainerStarted","Data":"218f8bf0452ce5f02bb7d60168a6a25bd4f3a0c42d071e4f2746315fd50cdb94"} Sep 07 10:42:18.317222 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.317167 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" podStartSLOduration=1.317151174 podStartE2EDuration="1.317151174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:42:18.314678465 +0000 UTC m=+1306.172367780" watchObservedRunningTime="2026-09-07 10:42:18.317151174 +0000 UTC m=+1306.174840488" Sep 07 10:42:18.801882 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:18.801839 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:19.807146 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:19.807119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:20.308879 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:20.308853 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:20.309662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:20.309644 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:37.252723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:37.252689 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:37.253296 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:37.252940 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" podUID="fe810b58-a843-4d04-ae70-5f73bc261329" containerName="istio-proxy" containerID="cri-o://3e22c28b23bc00b4b7a9e969e8868a06ffc362164373bfeb1725701dd039e961" gracePeriod=30 Sep 07 10:42:38.574032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.573991 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf"] Sep 07 10:42:38.578905 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.578883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:38.581065 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.581040 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:42:38.581170 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.581086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:42:38.585539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.585516 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf"] Sep 07 10:42:38.598753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.598727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf\" (UID: \"08fcc249-6099-4bc3-8caa-d2691c639912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:38.700156 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.700129 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf\" (UID: \"08fcc249-6099-4bc3-8caa-d2691c639912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:38.700692 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.700675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf\" (UID: \"08fcc249-6099-4bc3-8caa-d2691c639912\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:38.891603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:38.891496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:39.176164 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:39.176088 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf"] Sep 07 10:42:39.179228 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:42:39.179202 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fcc249_6099_4bc3_8caa_d2691c639912.slice/crio-8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10 WatchSource:0}: Error finding container 8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10: Status 404 returned error can't find the container with id 8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10 Sep 07 10:42:39.411713 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:39.411675 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" event={"ID":"08fcc249-6099-4bc3-8caa-d2691c639912","Type":"ContainerStarted","Data":"8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10"} Sep 07 10:42:40.417807 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:40.417701 2577 generic.go:356] "Generic (PLEG): container finished" podID="08fcc249-6099-4bc3-8caa-d2691c639912" containerID="3f5a298da83abe8adc55f6ca381bee2bcac6b7907d9b1f5736b9791b2c6d8784" exitCode=0 Sep 07 10:42:40.418160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:40.417794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" event={"ID":"08fcc249-6099-4bc3-8caa-d2691c639912","Type":"ContainerDied","Data":"3f5a298da83abe8adc55f6ca381bee2bcac6b7907d9b1f5736b9791b2c6d8784"} Sep 07 10:42:41.547193 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:41.547172 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:41.629526 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:41.629495 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca\") pod \"08fcc249-6099-4bc3-8caa-d2691c639912\" (UID: \"08fcc249-6099-4bc3-8caa-d2691c639912\") " Sep 07 10:42:41.629823 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:41.629803 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08fcc249-6099-4bc3-8caa-d2691c639912" (UID: "08fcc249-6099-4bc3-8caa-d2691c639912"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:41.730936 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:41.730908 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08fcc249-6099-4bc3-8caa-d2691c639912-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.432850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.432794 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" Sep 07 10:42:42.432850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.432833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf" event={"ID":"08fcc249-6099-4bc3-8caa-d2691c639912","Type":"ContainerDied","Data":"8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10"} Sep 07 10:42:42.433063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.432887 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8383f7b50215b098482c3eabd0d335852f4656c154164ea11fe7515084facd10" Sep 07 10:42:42.434842 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.434819 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe810b58-a843-4d04-ae70-5f73bc261329" containerID="3e22c28b23bc00b4b7a9e969e8868a06ffc362164373bfeb1725701dd039e961" exitCode=0 Sep 07 10:42:42.434968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.434853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" event={"ID":"fe810b58-a843-4d04-ae70-5f73bc261329","Type":"ContainerDied","Data":"3e22c28b23bc00b4b7a9e969e8868a06ffc362164373bfeb1725701dd039e961"} Sep 07 10:42:42.506196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.506179 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:42.537631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537561 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537636 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9z4c\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537691 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537746 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537771 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537794 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537841 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs\") pod \"fe810b58-a843-4d04-ae70-5f73bc261329\" (UID: \"fe810b58-a843-4d04-ae70-5f73bc261329\") " Sep 07 10:42:42.537996 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.537921 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:42:42.538221 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538097 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:42:42.538221 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538181 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:42:42.538324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538285 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:42:42.538540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538369 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:42.538540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538447 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.538540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538460 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.538540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538468 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.538540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.538475 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.540058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.540033 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c" (OuterVolumeSpecName: "kube-api-access-t9z4c") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "kube-api-access-t9z4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:42:42.540173 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.540160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:42:42.540440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.540417 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:42:42.540504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.540438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe810b58-a843-4d04-ae70-5f73bc261329" (UID: "fe810b58-a843-4d04-ae70-5f73bc261329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:42:42.639886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.639858 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe810b58-a843-4d04-ae70-5f73bc261329-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.639886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.639884 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe810b58-a843-4d04-ae70-5f73bc261329-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.639886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.639892 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.640281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.639900 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9z4c\" (UniqueName: \"kubernetes.io/projected/fe810b58-a843-4d04-ae70-5f73bc261329-kube-api-access-t9z4c\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:42.640281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:42.639909 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe810b58-a843-4d04-ae70-5f73bc261329-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:43.441662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:43.441633 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" Sep 07 10:42:43.441662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:43.441645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4" event={"ID":"fe810b58-a843-4d04-ae70-5f73bc261329","Type":"ContainerDied","Data":"218f8bf0452ce5f02bb7d60168a6a25bd4f3a0c42d071e4f2746315fd50cdb94"} Sep 07 10:42:43.441907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:43.441707 2577 scope.go:117] "RemoveContainer" containerID="3e22c28b23bc00b4b7a9e969e8868a06ffc362164373bfeb1725701dd039e961" Sep 07 10:42:43.458515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:43.458484 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:43.460964 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:43.460944 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-2b5f38-openshift-default-c7bb96856-hgzk4"] Sep 07 10:42:44.722728 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:44.722693 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe810b58-a843-4d04-ae70-5f73bc261329" path="/var/lib/kubelet/pods/fe810b58-a843-4d04-ae70-5f73bc261329/volumes" Sep 07 10:42:49.170239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.170201 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:42:49.170664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.170461 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" podUID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" containerName="payload-processing" containerID="cri-o://da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5" gracePeriod=30 Sep 07 10:42:49.179318 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.179293 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:42:49.179492 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.179474 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" podUID="5bbd1546-ddaf-4529-9335-12a93b89b064" containerName="payload-pre-processing" containerID="cri-o://67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87" gracePeriod=30 Sep 07 10:42:49.355121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.355086 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf"] Sep 07 10:42:49.357175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.357148 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ef687-4bbbf"] Sep 07 10:42:49.453985 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.453961 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:42:49.456646 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.456632 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:42:49.483054 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.483028 2577 generic.go:356] "Generic (PLEG): container finished" podID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" containerID="da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5" exitCode=0 Sep 07 10:42:49.483167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.483080 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" Sep 07 10:42:49.483167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.483100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" event={"ID":"2f7cc9b2-0667-4ada-9887-bf6a43f6f633","Type":"ContainerDied","Data":"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5"} Sep 07 10:42:49.483167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.483150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7" event={"ID":"2f7cc9b2-0667-4ada-9887-bf6a43f6f633","Type":"ContainerDied","Data":"27c7e5c871e5c8316990698b29625cbecba16406e0361230cd83a7911b18ab2a"} Sep 07 10:42:49.483347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.483181 2577 scope.go:117] "RemoveContainer" containerID="da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5" Sep 07 10:42:49.484463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.484443 2577 generic.go:356] "Generic (PLEG): container finished" podID="5bbd1546-ddaf-4529-9335-12a93b89b064" containerID="67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87" exitCode=0 Sep 07 10:42:49.484557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.484477 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" event={"ID":"5bbd1546-ddaf-4529-9335-12a93b89b064","Type":"ContainerDied","Data":"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87"} Sep 07 10:42:49.484557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.484485 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" Sep 07 10:42:49.484557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.484515 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt" event={"ID":"5bbd1546-ddaf-4529-9335-12a93b89b064","Type":"ContainerDied","Data":"def61d1eabd41b7ff525551047fc4c8acb34d18fa01f64ebf4a571a6d83f4e58"} Sep 07 10:42:49.493631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.493612 2577 scope.go:117] "RemoveContainer" containerID="da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5" Sep 07 10:42:49.494001 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:42:49.493972 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5\": container with ID starting with da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5 not found: ID does not exist" containerID="da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5" Sep 07 10:42:49.494084 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.494007 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5"} err="failed to get container status \"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5\": rpc error: code = NotFound desc = could not find container \"da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5\": container with ID starting with da94a01f3b3920213d9141e7e610d255a3e5b109f13a9c8696dad8df68a14ac5 not found: ID does not exist" Sep 07 10:42:49.494084 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.494029 2577 scope.go:117] "RemoveContainer" containerID="67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87" Sep 07 10:42:49.501420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.501397 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume\") pod \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " Sep 07 10:42:49.501552 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.501536 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume\") pod \"5bbd1546-ddaf-4529-9335-12a93b89b064\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " Sep 07 10:42:49.501635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.501573 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwrmq\" (UniqueName: \"kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq\") pod \"5bbd1546-ddaf-4529-9335-12a93b89b064\" (UID: \"5bbd1546-ddaf-4529-9335-12a93b89b064\") " Sep 07 10:42:49.501635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.501622 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4qcr\" (UniqueName: \"kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr\") pod \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\" (UID: \"2f7cc9b2-0667-4ada-9887-bf6a43f6f633\") " Sep 07 10:42:49.502064 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.501947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f7cc9b2-0667-4ada-9887-bf6a43f6f633" (UID: "2f7cc9b2-0667-4ada-9887-bf6a43f6f633"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:49.502255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.502235 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:49.502658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.502637 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5bbd1546-ddaf-4529-9335-12a93b89b064" (UID: "5bbd1546-ddaf-4529-9335-12a93b89b064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:49.502955 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.502859 2577 scope.go:117] "RemoveContainer" containerID="67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87" Sep 07 10:42:49.503182 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:42:49.503158 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87\": container with ID starting with 67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87 not found: ID does not exist" containerID="67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87" Sep 07 10:42:49.503267 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.503191 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87"} err="failed to get container status \"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87\": rpc error: code = NotFound desc = could not find container \"67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87\": container with ID starting with 67dcccb177508f154d728bacbad2e24da9f1a9768614454a6036b38497975a87 not found: ID does not exist" Sep 07 10:42:49.505734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.505700 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr" (OuterVolumeSpecName: "kube-api-access-c4qcr") pod "2f7cc9b2-0667-4ada-9887-bf6a43f6f633" (UID: "2f7cc9b2-0667-4ada-9887-bf6a43f6f633"). InnerVolumeSpecName "kube-api-access-c4qcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:42:49.506533 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.506472 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq" (OuterVolumeSpecName: "kube-api-access-kwrmq") pod "5bbd1546-ddaf-4529-9335-12a93b89b064" (UID: "5bbd1546-ddaf-4529-9335-12a93b89b064"). InnerVolumeSpecName "kube-api-access-kwrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:42:49.603281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.603249 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbd1546-ddaf-4529-9335-12a93b89b064-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:49.603281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.603277 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrmq\" (UniqueName: \"kubernetes.io/projected/5bbd1546-ddaf-4529-9335-12a93b89b064-kube-api-access-kwrmq\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:49.603281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.603290 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qcr\" (UniqueName: \"kubernetes.io/projected/2f7cc9b2-0667-4ada-9887-bf6a43f6f633-kube-api-access-c4qcr\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:42:49.814749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.814715 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:42:49.820992 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.820963 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ef687-667c6b5b99-b8xk7"] Sep 07 10:42:49.830767 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.830744 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:42:49.836260 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:49.836240 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1ef687-bdb6cdf94-sbtwt"] Sep 07 10:42:50.723822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:50.723787 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08fcc249-6099-4bc3-8caa-d2691c639912" path="/var/lib/kubelet/pods/08fcc249-6099-4bc3-8caa-d2691c639912/volumes" Sep 07 10:42:50.724191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:50.724147 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" path="/var/lib/kubelet/pods/2f7cc9b2-0667-4ada-9887-bf6a43f6f633/volumes" Sep 07 10:42:50.724452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:50.724439 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbd1546-ddaf-4529-9335-12a93b89b064" path="/var/lib/kubelet/pods/5bbd1546-ddaf-4529-9335-12a93b89b064/volumes" Sep 07 10:42:56.023132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.023051 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:42:56.023543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.023264 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" containerID="cri-o://5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e" gracePeriod=30 Sep 07 10:42:56.477776 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.477736 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw"] Sep 07 10:42:56.478258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478238 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbd1546-ddaf-4529-9335-12a93b89b064" containerName="payload-pre-processing" Sep 07 10:42:56.478258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478258 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbd1546-ddaf-4529-9335-12a93b89b064" containerName="payload-pre-processing" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478267 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08fcc249-6099-4bc3-8caa-d2691c639912" containerName="revoke-keys" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478274 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08fcc249-6099-4bc3-8caa-d2691c639912" containerName="revoke-keys" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478293 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe810b58-a843-4d04-ae70-5f73bc261329" containerName="istio-proxy" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478298 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe810b58-a843-4d04-ae70-5f73bc261329" containerName="istio-proxy" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478311 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" containerName="payload-processing" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478316 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" containerName="payload-processing" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478388 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe810b58-a843-4d04-ae70-5f73bc261329" containerName="istio-proxy" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478398 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbd1546-ddaf-4529-9335-12a93b89b064" containerName="payload-pre-processing" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478405 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f7cc9b2-0667-4ada-9887-bf6a43f6f633" containerName="payload-processing" Sep 07 10:42:56.478442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.478410 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08fcc249-6099-4bc3-8caa-d2691c639912" containerName="revoke-keys" Sep 07 10:42:56.482988 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.482967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:56.485073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.485052 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:42:56.485178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.485081 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:42:56.488390 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.488370 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw"] Sep 07 10:42:56.571473 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.571447 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw\" (UID: \"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:56.673017 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.672976 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw\" (UID: \"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:56.673646 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.673628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw\" (UID: \"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:56.796633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:56.796570 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:57.064031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:57.064009 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw"] Sep 07 10:42:57.066186 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:42:57.066164 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe43b5f8_138f_4d9f_9b68_1982a1b55b8f.slice/crio-d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922 WatchSource:0}: Error finding container d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922: Status 404 returned error can't find the container with id d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922 Sep 07 10:42:57.534133 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:57.534093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" event={"ID":"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f","Type":"ContainerStarted","Data":"5736024955bb7f8797ac6eaa36ea6b70520a2df83b3716d65f197b633047f1b8"} Sep 07 10:42:57.534277 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:57.534156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" event={"ID":"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f","Type":"ContainerStarted","Data":"d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922"} Sep 07 10:42:57.548140 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:57.548082 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" podStartSLOduration=1.548063055 podStartE2EDuration="1.548063055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:42:57.547894957 +0000 UTC m=+1345.405584275" watchObservedRunningTime="2026-09-07 10:42:57.548063055 +0000 UTC m=+1345.405752370" Sep 07 10:42:58.540777 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.540742 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" containerID="5736024955bb7f8797ac6eaa36ea6b70520a2df83b3716d65f197b633047f1b8" exitCode=0 Sep 07 10:42:58.541138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.540834 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" event={"ID":"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f","Type":"ContainerDied","Data":"5736024955bb7f8797ac6eaa36ea6b70520a2df83b3716d65f197b633047f1b8"} Sep 07 10:42:58.753735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.753701 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:42:58.758191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.758160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.760554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.760528 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-482359dc-openshift-default-dockercfg-g5xqd\"" Sep 07 10:42:58.767255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.767232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:42:58.894942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.894861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.894942 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.894915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895282 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jchx\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.895395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.895387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996344 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jchx\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996436 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996844 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996844 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996844 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996979 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996863 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.996979 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.997050 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.996980 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.997135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.997116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.998769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.998747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:58.999014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:58.998995 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:59.004855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.004828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:59.005021 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.005005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jchx\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx\") pod \"e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:59.073286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.073265 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:42:59.361714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.361689 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:42:59.363748 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:42:59.363724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3c5f38_6fa0_462e_9c40_740ad722a8d1.slice/crio-f755a6f7246fd7bb71baef5e315e9092de62df8c8ab723d14dd78555dee64fdd WatchSource:0}: Error finding container f755a6f7246fd7bb71baef5e315e9092de62df8c8ab723d14dd78555dee64fdd: Status 404 returned error can't find the container with id f755a6f7246fd7bb71baef5e315e9092de62df8c8ab723d14dd78555dee64fdd Sep 07 10:42:59.366124 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.366093 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:59.366210 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.366155 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:59.366210 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.366193 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:42:59.549574 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.549539 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerStarted","Data":"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e"} Sep 07 10:42:59.549999 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.549618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerStarted","Data":"f755a6f7246fd7bb71baef5e315e9092de62df8c8ab723d14dd78555dee64fdd"} Sep 07 10:42:59.622245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.622044 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podStartSLOduration=1.6220234 podStartE2EDuration="1.6220234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:42:59.618106586 +0000 UTC m=+1347.475795901" watchObservedRunningTime="2026-09-07 10:42:59.6220234 +0000 UTC m=+1347.479712717" Sep 07 10:42:59.670306 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.670280 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:42:59.812924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.812892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca\") pod \"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f\" (UID: \"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f\") " Sep 07 10:42:59.813317 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.813288 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" (UID: "fe43b5f8-138f-4d9f-9b68-1982a1b55b8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:42:59.914105 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:42:59.914028 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:00.073764 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.073724 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:00.078512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.078487 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:00.557341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.557313 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" Sep 07 10:43:00.557804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.557311 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw" event={"ID":"fe43b5f8-138f-4d9f-9b68-1982a1b55b8f","Type":"ContainerDied","Data":"d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922"} Sep 07 10:43:00.557804 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.557432 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58ec3f14401bd3fb41ff9bffcfafbb44a004dcc39e4b94b47f2b5efe2343922" Sep 07 10:43:00.557935 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.557916 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:00.558953 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:00.558932 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:01.275836 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.275816 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:43:01.427876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.427800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.427876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.427853 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smkdm\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.427892 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.427955 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428034 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:01.428098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428106 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428137 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428171 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428169 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:01.428297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428186 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data\") pod \"001cddd7-d6d0-459b-94d7-623348583a2f\" (UID: \"001cddd7-d6d0-459b-94d7-623348583a2f\") " Sep 07 10:43:01.428535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:01.428535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428508 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:01.428830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:01.428830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428716 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.428830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428737 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.428990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428819 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001cddd7-d6d0-459b-94d7-623348583a2f-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.428990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428919 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.428990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.428933 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.430300 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.430271 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm" (OuterVolumeSpecName: "kube-api-access-smkdm") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "kube-api-access-smkdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:01.430300 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.430284 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:01.430462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.430323 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:43:01.430462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.430376 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "001cddd7-d6d0-459b-94d7-623348583a2f" (UID: "001cddd7-d6d0-459b-94d7-623348583a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:01.530517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.530484 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.530517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.530509 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001cddd7-d6d0-459b-94d7-623348583a2f-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.530517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.530522 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkdm\" (UniqueName: \"kubernetes.io/projected/001cddd7-d6d0-459b-94d7-623348583a2f-kube-api-access-smkdm\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.530753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.530536 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001cddd7-d6d0-459b-94d7-623348583a2f-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:01.565756 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.565725 2577 generic.go:356] "Generic (PLEG): container finished" podID="001cddd7-d6d0-459b-94d7-623348583a2f" containerID="5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e" exitCode=0 Sep 07 10:43:01.566127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.565776 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" Sep 07 10:43:01.566127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.565808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerDied","Data":"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e"} Sep 07 10:43:01.566127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.565896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp" event={"ID":"001cddd7-d6d0-459b-94d7-623348583a2f","Type":"ContainerDied","Data":"e019881c9545fbb7260f8df8512e3e2043f621c557f2b2a5f2aaddb4e949538b"} Sep 07 10:43:01.566127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.565925 2577 scope.go:117] "RemoveContainer" containerID="5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e" Sep 07 10:43:01.576063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.576041 2577 scope.go:117] "RemoveContainer" containerID="cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f" Sep 07 10:43:01.585028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.585008 2577 scope.go:117] "RemoveContainer" containerID="5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e" Sep 07 10:43:01.585319 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:01.585302 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e\": container with ID starting with 5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e not found: ID does not exist" containerID="5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e" Sep 07 10:43:01.585380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.585328 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e"} err="failed to get container status \"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e\": rpc error: code = NotFound desc = could not find container \"5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e\": container with ID starting with 5354a8c954c390a7827776805b79572faa09b048a87e2ad0909911b86b2c071e not found: ID does not exist" Sep 07 10:43:01.585380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.585344 2577 scope.go:117] "RemoveContainer" containerID="cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f" Sep 07 10:43:01.585620 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:01.585575 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f\": container with ID starting with cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f not found: ID does not exist" containerID="cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f" Sep 07 10:43:01.585697 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.585624 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f"} err="failed to get container status \"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f\": rpc error: code = NotFound desc = could not find container \"cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f\": container with ID starting with cef1cf5ca7cea31275d4b9bbc31525381e2d5fc53dd30810179351a93d26472f not found: ID does not exist" Sep 07 10:43:01.597106 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.597087 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:43:01.611554 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:01.611530 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1ef687-openshift-default-d44b8cc68-64cjp"] Sep 07 10:43:02.724011 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:02.723975 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" path="/var/lib/kubelet/pods/001cddd7-d6d0-459b-94d7-623348583a2f/volumes" Sep 07 10:43:03.438046 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438012 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:03.438429 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438417 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" containerName="revoke-keys" Sep 07 10:43:03.438472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438432 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" containerName="revoke-keys" Sep 07 10:43:03.438472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438450 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.438472 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438455 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.438565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438474 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.438565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438480 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.438565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438548 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.438565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.438555 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" containerName="revoke-keys" Sep 07 10:43:03.443073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.443056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.445342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.445310 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-482359dc\"" Sep 07 10:43:03.445464 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.445390 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-482359dc-dockercfg-w82rl\"" Sep 07 10:43:03.449416 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.449388 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:03.479571 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.479541 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:03.480009 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.479996 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="001cddd7-d6d0-459b-94d7-623348583a2f" containerName="istio-proxy" Sep 07 10:43:03.482952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.482937 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.491908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.491886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:03.551091 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.551063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.551249 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.551117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5nc\" (UniqueName: \"kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.652483 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.652446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.652719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.652496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5nc\" (UniqueName: \"kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.652719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.652546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.652719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.652576 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kzt\" (UniqueName: \"kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.653236 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.653212 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.660674 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.660655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5nc\" (UniqueName: \"kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc\") pod \"payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.753223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.753202 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.753532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.753243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kzt\" (UniqueName: \"kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.753891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.753872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.755106 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.755088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:03.761388 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.761361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kzt\" (UniqueName: \"kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt\") pod \"payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:03.793742 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:03.793716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:04.033520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.033497 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:04.035987 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:04.035961 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6adb384_1e84_43d4_b1ee_4c677d9e0157.slice/crio-51ecb3b6ae2cd0ca2c6f5dc92116cc87b0ee8c97e4b4c9e5e5be9f1bc7aff124 WatchSource:0}: Error finding container 51ecb3b6ae2cd0ca2c6f5dc92116cc87b0ee8c97e4b4c9e5e5be9f1bc7aff124: Status 404 returned error can't find the container with id 51ecb3b6ae2cd0ca2c6f5dc92116cc87b0ee8c97e4b4c9e5e5be9f1bc7aff124 Sep 07 10:43:04.092627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.092601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:04.093217 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:04.093191 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9be6fd8_675d_427b_bfb3_440969208a2d.slice/crio-14f3d9895c1352088187236ed688ae8ac2b47aa3ead0d0e2a2a0a8f9884d749b WatchSource:0}: Error finding container 14f3d9895c1352088187236ed688ae8ac2b47aa3ead0d0e2a2a0a8f9884d749b: Status 404 returned error can't find the container with id 14f3d9895c1352088187236ed688ae8ac2b47aa3ead0d0e2a2a0a8f9884d749b Sep 07 10:43:04.585640 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.585576 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" event={"ID":"b9be6fd8-675d-427b-bfb3-440969208a2d","Type":"ContainerStarted","Data":"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e"} Sep 07 10:43:04.585837 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.585666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" event={"ID":"b9be6fd8-675d-427b-bfb3-440969208a2d","Type":"ContainerStarted","Data":"14f3d9895c1352088187236ed688ae8ac2b47aa3ead0d0e2a2a0a8f9884d749b"} Sep 07 10:43:04.585911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.585891 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:04.587467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.587440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" event={"ID":"b6adb384-1e84-43d4-b1ee-4c677d9e0157","Type":"ContainerStarted","Data":"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce"} Sep 07 10:43:04.587467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.587483 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" event={"ID":"b6adb384-1e84-43d4-b1ee-4c677d9e0157","Type":"ContainerStarted","Data":"51ecb3b6ae2cd0ca2c6f5dc92116cc87b0ee8c97e4b4c9e5e5be9f1bc7aff124"} Sep 07 10:43:04.587671 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.587521 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:04.602084 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:04.602035 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" podStartSLOduration=1.60202057 podStartE2EDuration="1.60202057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:04.600496867 +0000 UTC m=+1352.458186174" watchObservedRunningTime="2026-09-07 10:43:04.60202057 +0000 UTC m=+1352.459709883" Sep 07 10:43:07.468073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.468002 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" podStartSLOduration=4.467977758 podStartE2EDuration="4.467977758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:04.614202677 +0000 UTC m=+1352.471891992" watchObservedRunningTime="2026-09-07 10:43:07.467977758 +0000 UTC m=+1355.325667074" Sep 07 10:43:07.475736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.475699 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:43:07.475913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.475891 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" podUID="e04037e8-d806-4f47-a1d5-65796dd553e4" containerName="payload-processing" containerID="cri-o://947254ec46c7c63aeabf3284ef77fc84b0e496666db4b69bbc308c4c7cbece5c" gracePeriod=30 Sep 07 10:43:07.614994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.614832 2577 generic.go:356] "Generic (PLEG): container finished" podID="e04037e8-d806-4f47-a1d5-65796dd553e4" containerID="947254ec46c7c63aeabf3284ef77fc84b0e496666db4b69bbc308c4c7cbece5c" exitCode=0 Sep 07 10:43:07.614994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.614887 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" event={"ID":"e04037e8-d806-4f47-a1d5-65796dd553e4","Type":"ContainerDied","Data":"947254ec46c7c63aeabf3284ef77fc84b0e496666db4b69bbc308c4c7cbece5c"} Sep 07 10:43:07.677579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.677547 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw"] Sep 07 10:43:07.685631 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.685603 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b4061f-5cbdw"] Sep 07 10:43:07.741330 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.741308 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:43:07.894884 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.894849 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume\") pod \"e04037e8-d806-4f47-a1d5-65796dd553e4\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " Sep 07 10:43:07.895052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.894962 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznvr\" (UniqueName: \"kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr\") pod \"e04037e8-d806-4f47-a1d5-65796dd553e4\" (UID: \"e04037e8-d806-4f47-a1d5-65796dd553e4\") " Sep 07 10:43:07.895238 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.895213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e04037e8-d806-4f47-a1d5-65796dd553e4" (UID: "e04037e8-d806-4f47-a1d5-65796dd553e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:07.897075 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.897049 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr" (OuterVolumeSpecName: "kube-api-access-qznvr") pod "e04037e8-d806-4f47-a1d5-65796dd553e4" (UID: "e04037e8-d806-4f47-a1d5-65796dd553e4"). InnerVolumeSpecName "kube-api-access-qznvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:07.995959 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.995877 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04037e8-d806-4f47-a1d5-65796dd553e4-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:07.995959 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:07.995902 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznvr\" (UniqueName: \"kubernetes.io/projected/e04037e8-d806-4f47-a1d5-65796dd553e4-kube-api-access-qznvr\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:08.629461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.629414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" event={"ID":"e04037e8-d806-4f47-a1d5-65796dd553e4","Type":"ContainerDied","Data":"a63c08011aca5a91f2f5a545cbc2170d5ee07977a57d875b8fd74e12a17669a7"} Sep 07 10:43:08.630081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.629500 2577 scope.go:117] "RemoveContainer" containerID="947254ec46c7c63aeabf3284ef77fc84b0e496666db4b69bbc308c4c7cbece5c" Sep 07 10:43:08.630081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.629678 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf" Sep 07 10:43:08.657449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.657424 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:43:08.661144 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.661118 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b4061f-6fdf9595b5-npqtf"] Sep 07 10:43:08.731921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.731882 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e04037e8-d806-4f47-a1d5-65796dd553e4" path="/var/lib/kubelet/pods/e04037e8-d806-4f47-a1d5-65796dd553e4/volumes" Sep 07 10:43:08.732232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:08.732217 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe43b5f8-138f-4d9f-9b68-1982a1b55b8f" path="/var/lib/kubelet/pods/fe43b5f8-138f-4d9f-9b68-1982a1b55b8f/volumes" Sep 07 10:43:15.173920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.173883 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:43:15.174510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.174492 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04037e8-d806-4f47-a1d5-65796dd553e4" containerName="payload-processing" Sep 07 10:43:15.174510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.174513 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04037e8-d806-4f47-a1d5-65796dd553e4" containerName="payload-processing" Sep 07 10:43:15.174707 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.174638 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04037e8-d806-4f47-a1d5-65796dd553e4" containerName="payload-processing" Sep 07 10:43:15.179614 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.179579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.184102 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.184081 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5a0ff8-openshift-default-dockercfg-5ndms\"" Sep 07 10:43:15.208440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.208360 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:43:15.275247 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275340 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.275724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.275667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9lw\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376311 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376451 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376476 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376545 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9lw\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376682 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.376965 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376885 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.377228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.376992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.377228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.377199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.377618 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.377557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.378932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.378906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.379033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.378948 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.385244 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.385032 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.385362 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.385346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9lw\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw\") pod \"e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.490019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.489999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:15.594940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.594911 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:15.595072 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.594989 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:15.768915 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.768886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:43:15.771370 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:15.771336 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb09564_22c0_4614_bec1_c4ed18543234.slice/crio-c44c694ec122be3e83ef00a7fd95d4a109bed3617e66ca94d189c2967b974b90 WatchSource:0}: Error finding container c44c694ec122be3e83ef00a7fd95d4a109bed3617e66ca94d189c2967b974b90: Status 404 returned error can't find the container with id c44c694ec122be3e83ef00a7fd95d4a109bed3617e66ca94d189c2967b974b90 Sep 07 10:43:15.774296 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.774256 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:15.774394 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.774344 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:15.774452 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:15.774394 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:16.304913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.304143 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:54268->10.134.0.58:15021: read: connection reset by peer" start-of-body= Sep 07 10:43:16.304913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.304205 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:54268->10.134.0.58:15021: read: connection reset by peer" Sep 07 10:43:16.678488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.678403 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerID="a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e" exitCode=0 Sep 07 10:43:16.678488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.678476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerDied","Data":"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e"} Sep 07 10:43:16.678963 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.678943 2577 scope.go:117] "RemoveContainer" containerID="a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e" Sep 07 10:43:16.680061 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.680031 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:16.680170 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.680106 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:16.680170 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.680148 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:16.680246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.680194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerStarted","Data":"55bd0aa8bf7237a2c5f34fe1a04924b95d99b0f80fce165049502260657a0969"} Sep 07 10:43:16.680246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:16.680231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerStarted","Data":"c44c694ec122be3e83ef00a7fd95d4a109bed3617e66ca94d189c2967b974b90"} Sep 07 10:43:17.491151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.491115 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:17.496172 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.496145 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:17.516089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.516026 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podStartSLOduration=2.5160041570000002 podStartE2EDuration="2.516004157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:16.715452477 +0000 UTC m=+1364.573141794" watchObservedRunningTime="2026-09-07 10:43:17.516004157 +0000 UTC m=+1365.373693475" Sep 07 10:43:17.687476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.687442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerStarted","Data":"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6"} Sep 07 10:43:17.687762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.687731 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:17.688926 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:17.688904 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:18.074135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:18.074093 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:18.074342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:18.074314 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 07 10:43:18.074393 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:18.074372 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 07 10:43:19.073850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:19.073816 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 07 10:43:19.074246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:19.073870 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 07 10:43:20.073960 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:20.073918 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 07 10:43:20.074375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:20.073977 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 07 10:43:21.074513 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:21.074475 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 07 10:43:21.074958 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:21.074540 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 07 10:43:22.073818 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:22.073775 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 07 10:43:22.074019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:22.073839 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 07 10:43:23.077313 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:23.077285 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:23.077738 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:23.077545 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:23.078244 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:23.078224 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:32.689321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.689224 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:32.689321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.689291 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:32.784034 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.784004 2577 generic.go:356] "Generic (PLEG): container finished" podID="1cb09564-22c0-4614-bec1-c4ed18543234" containerID="55bd0aa8bf7237a2c5f34fe1a04924b95d99b0f80fce165049502260657a0969" exitCode=0 Sep 07 10:43:32.784225 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.784087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerDied","Data":"55bd0aa8bf7237a2c5f34fe1a04924b95d99b0f80fce165049502260657a0969"} Sep 07 10:43:32.784478 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.784463 2577 scope.go:117] "RemoveContainer" containerID="55bd0aa8bf7237a2c5f34fe1a04924b95d99b0f80fce165049502260657a0969" Sep 07 10:43:32.785466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.785439 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:32.785530 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.785503 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:32.785566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:32.785537 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:33.791647 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:33.791614 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerStarted","Data":"176d41111cc97df5aee0221ed8ea07cf800bd9e7197b7247557b9bf594fad9a6"} Sep 07 10:43:34.271283 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.271246 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh"] Sep 07 10:43:34.274041 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.274020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:34.276113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.276088 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:43:34.276223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.276110 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:43:34.280933 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.280913 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh"] Sep 07 10:43:34.385763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.385730 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-482359dc-vxwxh\" (UID: \"8643f770-f9c5-4689-8578-933743186201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:34.487249 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.487212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-482359dc-vxwxh\" (UID: \"8643f770-f9c5-4689-8578-933743186201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:34.487848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.487830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-482359dc-vxwxh\" (UID: \"8643f770-f9c5-4689-8578-933743186201\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:34.491092 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.491072 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:34.491254 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.491237 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:34.491298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.491281 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:34.586535 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.586450 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:34.885711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:34.885629 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh"] Sep 07 10:43:34.888468 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:34.888441 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8643f770_f9c5_4689_8578_933743186201.slice/crio-c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0 WatchSource:0}: Error finding container c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0: Status 404 returned error can't find the container with id c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0 Sep 07 10:43:35.491217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:35.491181 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:35.491368 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:35.491245 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:35.803424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:35.803386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" event={"ID":"8643f770-f9c5-4689-8578-933743186201","Type":"ContainerStarted","Data":"124e510c038ff2f0e54f9e0e3c33acb7af21c99b6943fae1bbd998f745b2170e"} Sep 07 10:43:35.803629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:35.803439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" event={"ID":"8643f770-f9c5-4689-8578-933743186201","Type":"ContainerStarted","Data":"c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0"} Sep 07 10:43:35.820042 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:35.819970 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" podStartSLOduration=1.819950097 podStartE2EDuration="1.819950097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:35.818742595 +0000 UTC m=+1383.676431913" watchObservedRunningTime="2026-09-07 10:43:35.819950097 +0000 UTC m=+1383.677639413" Sep 07 10:43:36.490934 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:36.490894 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:36.491367 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:36.490988 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:36.811938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:36.811841 2577 generic.go:356] "Generic (PLEG): container finished" podID="8643f770-f9c5-4689-8578-933743186201" containerID="124e510c038ff2f0e54f9e0e3c33acb7af21c99b6943fae1bbd998f745b2170e" exitCode=0 Sep 07 10:43:36.811938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:36.811910 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" event={"ID":"8643f770-f9c5-4689-8578-933743186201","Type":"ContainerDied","Data":"124e510c038ff2f0e54f9e0e3c33acb7af21c99b6943fae1bbd998f745b2170e"} Sep 07 10:43:37.490928 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:37.490895 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:37.491108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:37.490945 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:37.977898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:37.977875 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:38.137177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.137074 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca\") pod \"8643f770-f9c5-4689-8578-933743186201\" (UID: \"8643f770-f9c5-4689-8578-933743186201\") " Sep 07 10:43:38.137451 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.137426 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8643f770-f9c5-4689-8578-933743186201" (UID: "8643f770-f9c5-4689-8578-933743186201"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:38.238930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.238900 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8643f770-f9c5-4689-8578-933743186201-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:38.491159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.491123 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 07 10:43:38.491529 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.491179 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 07 10:43:38.827013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.826917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" event={"ID":"8643f770-f9c5-4689-8578-933743186201","Type":"ContainerDied","Data":"c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0"} Sep 07 10:43:38.827013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.826949 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh" Sep 07 10:43:38.827013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:38.826973 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c118f92f25e3ff940588874b40d151c434b0703b06eb02d71d2fc75909cf00a0" Sep 07 10:43:39.494850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:39.494816 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:39.495271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:39.495065 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:39.495784 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:39.495768 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:43:44.507322 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.507293 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:43:44.507781 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.507762 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8643f770-f9c5-4689-8578-933743186201" containerName="revoke-keys" Sep 07 10:43:44.507898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.507788 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8643f770-f9c5-4689-8578-933743186201" containerName="revoke-keys" Sep 07 10:43:44.507898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.507861 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8643f770-f9c5-4689-8578-933743186201" containerName="revoke-keys" Sep 07 10:43:44.511031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.511012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.513187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.513167 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-659410-openshift-default-dockercfg-dkmbd\"" Sep 07 10:43:44.523322 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.523300 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:43:44.600359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600544 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb4zx\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.600930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.600874 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701399 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701489 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb4zx\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.701972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701848 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.702108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.701965 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.702108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.702030 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.702255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.702229 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.703796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.703780 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.704668 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.704644 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.714264 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.714245 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb4zx\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.714369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.714299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token\") pod \"e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:44.822110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:44.822035 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:45.106399 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.105302 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:43:45.108162 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:45.108111 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d88a4e_9c01_4e74_8274_af54b8a47cbe.slice/crio-69049a81afe9e9024f1f01f7d80dc4bc029232be375c7902f45075ddab65a62f WatchSource:0}: Error finding container 69049a81afe9e9024f1f01f7d80dc4bc029232be375c7902f45075ddab65a62f: Status 404 returned error can't find the container with id 69049a81afe9e9024f1f01f7d80dc4bc029232be375c7902f45075ddab65a62f Sep 07 10:43:45.110379 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.110342 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:45.110463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.110427 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:45.110503 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.110469 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:43:45.159823 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.159793 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:45.160285 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.160078 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" podUID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" containerName="payload-processing" containerID="cri-o://5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce" gracePeriod=30 Sep 07 10:43:45.169489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.169468 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:45.169778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.169749 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" podUID="b9be6fd8-675d-427b-bfb3-440969208a2d" containerName="payload-pre-processing" containerID="cri-o://697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e" gracePeriod=30 Sep 07 10:43:45.333699 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.333635 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh"] Sep 07 10:43:45.336145 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.336116 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-482359dc-vxwxh"] Sep 07 10:43:45.440350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.440323 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:45.451845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.451827 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:45.612109 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume\") pod \"b9be6fd8-675d-427b-bfb3-440969208a2d\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " Sep 07 10:43:45.612515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612165 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume\") pod \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " Sep 07 10:43:45.612515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612238 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kzt\" (UniqueName: \"kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt\") pod \"b9be6fd8-675d-427b-bfb3-440969208a2d\" (UID: \"b9be6fd8-675d-427b-bfb3-440969208a2d\") " Sep 07 10:43:45.612515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612267 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv5nc\" (UniqueName: \"kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc\") pod \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\" (UID: \"b6adb384-1e84-43d4-b1ee-4c677d9e0157\") " Sep 07 10:43:45.612515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612461 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9be6fd8-675d-427b-bfb3-440969208a2d" (UID: "b9be6fd8-675d-427b-bfb3-440969208a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:45.612744 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612541 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6adb384-1e84-43d4-b1ee-4c677d9e0157" (UID: "b6adb384-1e84-43d4-b1ee-4c677d9e0157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:45.612744 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612688 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9be6fd8-675d-427b-bfb3-440969208a2d-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:45.612744 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.612706 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6adb384-1e84-43d4-b1ee-4c677d9e0157-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:45.614431 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.614409 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt" (OuterVolumeSpecName: "kube-api-access-b6kzt") pod "b9be6fd8-675d-427b-bfb3-440969208a2d" (UID: "b9be6fd8-675d-427b-bfb3-440969208a2d"). InnerVolumeSpecName "kube-api-access-b6kzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:45.614514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.614438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc" (OuterVolumeSpecName: "kube-api-access-gv5nc") pod "b6adb384-1e84-43d4-b1ee-4c677d9e0157" (UID: "b6adb384-1e84-43d4-b1ee-4c677d9e0157"). InnerVolumeSpecName "kube-api-access-gv5nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:45.713688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.713662 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kzt\" (UniqueName: \"kubernetes.io/projected/b9be6fd8-675d-427b-bfb3-440969208a2d-kube-api-access-b6kzt\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:45.713785 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.713692 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5nc\" (UniqueName: \"kubernetes.io/projected/b6adb384-1e84-43d4-b1ee-4c677d9e0157-kube-api-access-gv5nc\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:45.869471 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.869438 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerStarted","Data":"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f"} Sep 07 10:43:45.869687 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.869495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerStarted","Data":"69049a81afe9e9024f1f01f7d80dc4bc029232be375c7902f45075ddab65a62f"} Sep 07 10:43:45.870867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.870842 2577 generic.go:356] "Generic (PLEG): container finished" podID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" containerID="5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce" exitCode=0 Sep 07 10:43:45.871000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.870873 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" Sep 07 10:43:45.871000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.870922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" event={"ID":"b6adb384-1e84-43d4-b1ee-4c677d9e0157","Type":"ContainerDied","Data":"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce"} Sep 07 10:43:45.871000 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.870983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2" event={"ID":"b6adb384-1e84-43d4-b1ee-4c677d9e0157","Type":"ContainerDied","Data":"51ecb3b6ae2cd0ca2c6f5dc92116cc87b0ee8c97e4b4c9e5e5be9f1bc7aff124"} Sep 07 10:43:45.871170 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.871014 2577 scope.go:117] "RemoveContainer" containerID="5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce" Sep 07 10:43:45.879024 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.878993 2577 generic.go:356] "Generic (PLEG): container finished" podID="b9be6fd8-675d-427b-bfb3-440969208a2d" containerID="697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e" exitCode=0 Sep 07 10:43:45.879143 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.879049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" event={"ID":"b9be6fd8-675d-427b-bfb3-440969208a2d","Type":"ContainerDied","Data":"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e"} Sep 07 10:43:45.879143 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.879135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" event={"ID":"b9be6fd8-675d-427b-bfb3-440969208a2d","Type":"ContainerDied","Data":"14f3d9895c1352088187236ed688ae8ac2b47aa3ead0d0e2a2a0a8f9884d749b"} Sep 07 10:43:45.879290 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.879269 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr" Sep 07 10:43:45.884242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.884222 2577 scope.go:117] "RemoveContainer" containerID="5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce" Sep 07 10:43:45.884554 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:45.884524 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce\": container with ID starting with 5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce not found: ID does not exist" containerID="5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce" Sep 07 10:43:45.884664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.884564 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce"} err="failed to get container status \"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce\": rpc error: code = NotFound desc = could not find container \"5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce\": container with ID starting with 5ab98f876e61e57a012322ab2764852e78d98ff1da259599cd55ff77e11f65ce not found: ID does not exist" Sep 07 10:43:45.884664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.884617 2577 scope.go:117] "RemoveContainer" containerID="697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e" Sep 07 10:43:45.892049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.892010 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podStartSLOduration=1.8919966540000002 podStartE2EDuration="1.891996654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:45.888851139 +0000 UTC m=+1393.746540453" watchObservedRunningTime="2026-09-07 10:43:45.891996654 +0000 UTC m=+1393.749685967" Sep 07 10:43:45.895069 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.895048 2577 scope.go:117] "RemoveContainer" containerID="697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e" Sep 07 10:43:45.895398 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:45.895377 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e\": container with ID starting with 697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e not found: ID does not exist" containerID="697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e" Sep 07 10:43:45.895489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.895406 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e"} err="failed to get container status \"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e\": rpc error: code = NotFound desc = could not find container \"697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e\": container with ID starting with 697a2650c8af54baea2d2a21030c29fad67c977dbe48923b067708cbed703e7e not found: ID does not exist" Sep 07 10:43:45.904873 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.904854 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:45.910708 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.910688 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-482359dc-6c5ffc9469-z2zd2"] Sep 07 10:43:45.919634 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.919616 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:45.926235 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:45.926216 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-482359dc-59df8bffb6-hwxmr"] Sep 07 10:43:46.723456 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.723426 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8643f770-f9c5-4689-8578-933743186201" path="/var/lib/kubelet/pods/8643f770-f9c5-4689-8578-933743186201/volumes" Sep 07 10:43:46.723818 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.723773 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" path="/var/lib/kubelet/pods/b6adb384-1e84-43d4-b1ee-4c677d9e0157/volumes" Sep 07 10:43:46.724073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.724061 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9be6fd8-675d-427b-bfb3-440969208a2d" path="/var/lib/kubelet/pods/b9be6fd8-675d-427b-bfb3-440969208a2d/volumes" Sep 07 10:43:46.822745 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.822712 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:46.827908 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.827885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:46.886223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.886201 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:46.887082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:46.887052 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:43:49.691968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.691932 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:43:49.692435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692370 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9be6fd8-675d-427b-bfb3-440969208a2d" containerName="payload-pre-processing" Sep 07 10:43:49.692435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692381 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9be6fd8-675d-427b-bfb3-440969208a2d" containerName="payload-pre-processing" Sep 07 10:43:49.692435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692405 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" containerName="payload-processing" Sep 07 10:43:49.692435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692411 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" containerName="payload-processing" Sep 07 10:43:49.692621 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692468 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9be6fd8-675d-427b-bfb3-440969208a2d" containerName="payload-pre-processing" Sep 07 10:43:49.692621 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.692482 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6adb384-1e84-43d4-b1ee-4c677d9e0157" containerName="payload-processing" Sep 07 10:43:49.696845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.696829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.699550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.699517 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-659410-dockercfg-47sjp\"" Sep 07 10:43:49.699961 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.699942 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-659410\"" Sep 07 10:43:49.710816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.710792 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:43:49.715904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.715880 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:43:49.719773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.719749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.730645 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.730622 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:43:49.749073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.749046 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.749215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.749099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.749215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.749184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnm9b\" (UniqueName: \"kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.749333 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.749319 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwsx\" (UniqueName: \"kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.849979 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.849932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.850151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.849984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnm9b\" (UniqueName: \"kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.850151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.850101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwsx\" (UniqueName: \"kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.850228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.850193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.850772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.850747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:49.851006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.850985 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.857935 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.857906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwsx\" (UniqueName: \"kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx\") pod \"payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:49.857935 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:49.857929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnm9b\" (UniqueName: \"kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b\") pod \"payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:50.008239 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.008210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:50.030438 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.030411 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:50.285216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.285183 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:43:50.286766 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:50.286740 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5f75ba_1f25_43d2_930c_1e6b614ee80d.slice/crio-241d3a53ef1ffce3020e97b630cb41ce8646b92e0405ff06de7e80dd7a0a55ee WatchSource:0}: Error finding container 241d3a53ef1ffce3020e97b630cb41ce8646b92e0405ff06de7e80dd7a0a55ee: Status 404 returned error can't find the container with id 241d3a53ef1ffce3020e97b630cb41ce8646b92e0405ff06de7e80dd7a0a55ee Sep 07 10:43:50.314504 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.314479 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:43:50.320732 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:43:50.320708 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f44ca8_36be_4f74_ba35_7588bf735352.slice/crio-3df2419c50ae2b82dfc28ff25fa14b963e118cdba70b842c23bdec154da7ba7c WatchSource:0}: Error finding container 3df2419c50ae2b82dfc28ff25fa14b963e118cdba70b842c23bdec154da7ba7c: Status 404 returned error can't find the container with id 3df2419c50ae2b82dfc28ff25fa14b963e118cdba70b842c23bdec154da7ba7c Sep 07 10:43:50.911714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.911673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" event={"ID":"6b5f75ba-1f25-43d2-930c-1e6b614ee80d","Type":"ContainerStarted","Data":"bfbed057a8b05510edea221a23aa3b3e958545cf16f43c6bfd5dc5b5a242aced"} Sep 07 10:43:50.912207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.911740 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" event={"ID":"6b5f75ba-1f25-43d2-930c-1e6b614ee80d","Type":"ContainerStarted","Data":"241d3a53ef1ffce3020e97b630cb41ce8646b92e0405ff06de7e80dd7a0a55ee"} Sep 07 10:43:50.912207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.911793 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:43:50.913507 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.913472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" event={"ID":"c7f44ca8-36be-4f74-ba35-7588bf735352","Type":"ContainerStarted","Data":"25a39d8f7d670dab67f2de3835f52780dd4b8f81b005622d19a32431b3145727"} Sep 07 10:43:50.913657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.913519 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:43:50.913657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.913536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" event={"ID":"c7f44ca8-36be-4f74-ba35-7588bf735352","Type":"ContainerStarted","Data":"3df2419c50ae2b82dfc28ff25fa14b963e118cdba70b842c23bdec154da7ba7c"} Sep 07 10:43:50.929346 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.929292 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" podStartSLOduration=1.929265967 podStartE2EDuration="1.929265967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:50.9279428 +0000 UTC m=+1398.785632140" watchObservedRunningTime="2026-09-07 10:43:50.929265967 +0000 UTC m=+1398.786955283" Sep 07 10:43:50.946027 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:50.945964 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" podStartSLOduration=1.945940821 podStartE2EDuration="1.945940821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:43:50.944181225 +0000 UTC m=+1398.801870544" watchObservedRunningTime="2026-09-07 10:43:50.945940821 +0000 UTC m=+1398.803630138" Sep 07 10:43:51.978327 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:51.978295 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:43:51.979036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:51.979005 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" containerID="cri-o://d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6" gracePeriod=30 Sep 07 10:43:53.079702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:53.079666 2577 patch_prober.go:28] interesting pod/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 07 10:43:53.080057 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:53.079723 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:43:57.264328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.264300 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:57.337045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337011 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337058 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jchx\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337114 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337132 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337188 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337205 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337250 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337335 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:57.337688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337365 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert\") pod \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\" (UID: \"ae3c5f38-6fa0-462e-9c40-740ad722a8d1\") " Sep 07 10:43:57.337688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337404 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:57.337688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337487 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:57.337688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337533 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:57.337920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337797 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.337920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337817 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.337920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337832 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.337920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337845 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.337920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.337797 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:43:57.339729 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.339700 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:57.339848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.339753 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:43:57.340059 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.340039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:43:57.340111 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.340042 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx" (OuterVolumeSpecName: "kube-api-access-8jchx") pod "ae3c5f38-6fa0-462e-9c40-740ad722a8d1" (UID: "ae3c5f38-6fa0-462e-9c40-740ad722a8d1"). InnerVolumeSpecName "kube-api-access-8jchx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:43:57.439063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.439027 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.439063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.439055 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.439063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.439064 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jchx\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-kube-api-access-8jchx\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.439063 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.439074 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.439330 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.439084 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c5f38-6fa0-462e-9c40-740ad722a8d1-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:43:57.956096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.956064 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerID="d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6" exitCode=0 Sep 07 10:43:57.956281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.956115 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" Sep 07 10:43:57.956281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.956152 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerDied","Data":"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6"} Sep 07 10:43:57.956281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.956218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l" event={"ID":"ae3c5f38-6fa0-462e-9c40-740ad722a8d1","Type":"ContainerDied","Data":"f755a6f7246fd7bb71baef5e315e9092de62df8c8ab723d14dd78555dee64fdd"} Sep 07 10:43:57.956281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.956249 2577 scope.go:117] "RemoveContainer" containerID="d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6" Sep 07 10:43:57.965871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.965853 2577 scope.go:117] "RemoveContainer" containerID="a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e" Sep 07 10:43:57.974106 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.974077 2577 scope.go:117] "RemoveContainer" containerID="d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6" Sep 07 10:43:57.974566 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:57.974447 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6\": container with ID starting with d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6 not found: ID does not exist" containerID="d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6" Sep 07 10:43:57.974566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.974483 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6"} err="failed to get container status \"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6\": rpc error: code = NotFound desc = could not find container \"d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6\": container with ID starting with d81838e4b039867b5a4f994c4e9efa2b3feecaf062c1ce32df362eed9f4fccb6 not found: ID does not exist" Sep 07 10:43:57.974566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.974504 2577 scope.go:117] "RemoveContainer" containerID="a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e" Sep 07 10:43:57.976104 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:43:57.975160 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e\": container with ID starting with a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e not found: ID does not exist" containerID="a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e" Sep 07 10:43:57.976104 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.975192 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e"} err="failed to get container status \"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e\": rpc error: code = NotFound desc = could not find container \"a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e\": container with ID starting with a8e16fd6f631e93e6b2a90d728d2130335da2fc7b52b6d4a359bb32937b91c3e not found: ID does not exist" Sep 07 10:43:57.977183 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.977163 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:43:57.981314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:57.981293 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-482359dc-openshift-default-7849f456d8-bkx7l"] Sep 07 10:43:58.727775 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:43:58.727732 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" path="/var/lib/kubelet/pods/ae3c5f38-6fa0-462e-9c40-740ad722a8d1/volumes" Sep 07 10:44:01.922051 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:01.922017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:44:01.922502 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:01.922094 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:44:02.928816 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:02.928761 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:44:02.929341 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:02.928859 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:44:05.000488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.000451 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerID="1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f" exitCode=0 Sep 07 10:44:05.000919 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.000511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerDied","Data":"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f"} Sep 07 10:44:05.000977 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.000948 2577 scope.go:117] "RemoveContainer" containerID="1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f" Sep 07 10:44:05.002032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.001988 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:05.002132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.002069 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:05.002132 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:05.002109 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:06.036381 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:06.035451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerStarted","Data":"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d"} Sep 07 10:44:06.822735 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:06.822635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:44:06.822954 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:06.822898 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:44:06.823032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:06.822957 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 07 10:44:07.823109 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:07.823069 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:44:07.823723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:07.823688 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 07 10:44:08.822711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:08.822670 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:44:08.822927 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:08.822736 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 07 10:44:09.823123 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:09.823083 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:44:09.823754 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:09.823717 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 07 10:44:10.823788 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:10.822673 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Sep 07 10:44:10.823788 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:10.822741 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Sep 07 10:44:11.459346 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459312 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:44:11.459803 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459789 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:11.459803 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459804 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:11.459913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459812 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:11.459913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459817 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:11.459913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.459885 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:11.461978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.461958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.464802 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.464781 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-a158df\"" Sep 07 10:44:11.464904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.464892 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-a158df-dockercfg-8tlq5\"" Sep 07 10:44:11.474951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.474929 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:44:11.595626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.595572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.595798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.595643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzpfv\" (UniqueName: \"kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.696980 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.696942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.697168 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.696989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzpfv\" (UniqueName: \"kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.697692 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.697672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.704886 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.704860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzpfv\" (UniqueName: \"kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv\") pod \"payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.772556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.772515 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:11.828370 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.828336 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:44:11.829384 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.829309 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:44:11.830693 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:11.830657 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:44:12.045115 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:12.045089 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:44:12.047019 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:44:12.046979 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3ff872_6409_46cd_990f_68a60632fdbe.slice/crio-4371c1216acc0dfed20d8d59dd487a7f144ee7b249f3c4036f29eb86f8174e26 WatchSource:0}: Error finding container 4371c1216acc0dfed20d8d59dd487a7f144ee7b249f3c4036f29eb86f8174e26: Status 404 returned error can't find the container with id 4371c1216acc0dfed20d8d59dd487a7f144ee7b249f3c4036f29eb86f8174e26 Sep 07 10:44:12.070772 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:12.070746 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" event={"ID":"5a3ff872-6409-46cd-990f-68a60632fdbe","Type":"ContainerStarted","Data":"4371c1216acc0dfed20d8d59dd487a7f144ee7b249f3c4036f29eb86f8174e26"} Sep 07 10:44:13.085283 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:13.084201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" event={"ID":"5a3ff872-6409-46cd-990f-68a60632fdbe","Type":"ContainerStarted","Data":"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944"} Sep 07 10:44:13.085283 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:13.084469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:13.102332 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:13.102286 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" podStartSLOduration=2.102265214 podStartE2EDuration="2.102265214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:13.099575303 +0000 UTC m=+1420.957264619" watchObservedRunningTime="2026-09-07 10:44:13.102265214 +0000 UTC m=+1420.959954530" Sep 07 10:44:24.092562 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:24.092482 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:44:34.619026 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.618985 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:44:34.619859 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.619836 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae3c5f38-6fa0-462e-9c40-740ad722a8d1" containerName="istio-proxy" Sep 07 10:44:34.624773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.624754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.626961 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.626942 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5e934f-openshift-default-dockercfg-8hsx2\"" Sep 07 10:44:34.633844 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.633819 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:44:34.719922 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.719884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.719997 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720121 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720108 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720249 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjsln\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720249 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720266 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720347 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.720624 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.720382 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.821990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.821951 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjsln\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822187 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822507 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822771 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822833 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.822833 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.822785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.824629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.824605 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.824976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.824953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.829848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.829828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.830089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.830069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjsln\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln\") pod \"e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:34.940058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:34.939972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:35.221145 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:35.221117 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:44:35.223445 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:44:35.223420 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74011a9d_e056_4bed_8b3e_bc1703cfca72.slice/crio-9f7bcc158033c2fe764bbfd464c23fd43bbbee681bb168485b43d408f5dcf4ca WatchSource:0}: Error finding container 9f7bcc158033c2fe764bbfd464c23fd43bbbee681bb168485b43d408f5dcf4ca: Status 404 returned error can't find the container with id 9f7bcc158033c2fe764bbfd464c23fd43bbbee681bb168485b43d408f5dcf4ca Sep 07 10:44:35.226135 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:35.226101 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:35.226217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:35.226175 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:35.226217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:35.226204 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:36.220627 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:36.220574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerStarted","Data":"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3"} Sep 07 10:44:36.221023 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:36.220648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerStarted","Data":"9f7bcc158033c2fe764bbfd464c23fd43bbbee681bb168485b43d408f5dcf4ca"} Sep 07 10:44:36.241749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:36.241703 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podStartSLOduration=2.241688711 podStartE2EDuration="2.241688711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:36.239018138 +0000 UTC m=+1444.096707467" watchObservedRunningTime="2026-09-07 10:44:36.241688711 +0000 UTC m=+1444.099378027" Sep 07 10:44:36.941243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:36.941205 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:36.946566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:36.946537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:37.226732 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:37.226693 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:37.228193 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:37.228173 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:40.091656 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.091620 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:44:40.095469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.095454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.098034 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.098013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5e934f\"" Sep 07 10:44:40.105725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.105700 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:44:40.178760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.178732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdxth\" (UniqueName: \"kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.178878 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.178794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.279673 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.279635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.279861 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.279760 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdxth\" (UniqueName: \"kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.280408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.280382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.287369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.287345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdxth\" (UniqueName: \"kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth\") pod \"payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.408361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.408281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:40.681178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:40.681153 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:44:40.683428 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:44:40.683397 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b16f0b_f8e3_4660_9284_cb40f4dc461b.slice/crio-fc78a3ddfc6f147c1b721bce2beba5e7a3c2a41b754469658df0f649398e6ab6 WatchSource:0}: Error finding container fc78a3ddfc6f147c1b721bce2beba5e7a3c2a41b754469658df0f649398e6ab6: Status 404 returned error can't find the container with id fc78a3ddfc6f147c1b721bce2beba5e7a3c2a41b754469658df0f649398e6ab6 Sep 07 10:44:41.257013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:41.256947 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" event={"ID":"a0b16f0b-f8e3-4660-9284-cb40f4dc461b","Type":"ContainerStarted","Data":"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8"} Sep 07 10:44:41.257013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:41.257017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" event={"ID":"a0b16f0b-f8e3-4660-9284-cb40f4dc461b","Type":"ContainerStarted","Data":"fc78a3ddfc6f147c1b721bce2beba5e7a3c2a41b754469658df0f649398e6ab6"} Sep 07 10:44:41.257557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:41.257049 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:41.272762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:41.272718 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" podStartSLOduration=1.272700698 podStartE2EDuration="1.272700698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:44:41.271367365 +0000 UTC m=+1449.129056681" watchObservedRunningTime="2026-09-07 10:44:41.272700698 +0000 UTC m=+1449.130390020" Sep 07 10:44:52.267498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:52.267469 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:44:58.374675 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.374641 2577 generic.go:356] "Generic (PLEG): container finished" podID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerID="1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3" exitCode=0 Sep 07 10:44:58.375328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.374722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerDied","Data":"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3"} Sep 07 10:44:58.375328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.375171 2577 scope.go:117] "RemoveContainer" containerID="1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3" Sep 07 10:44:58.376229 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.376204 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:58.376286 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.376271 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:58.376326 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:58.376301 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:44:59.382669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:59.382630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerStarted","Data":"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14"} Sep 07 10:44:59.940289 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:59.940256 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:44:59.940494 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:59.940468 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 07 10:44:59.940572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:44:59.940543 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 07 10:45:00.147750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.147673 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n"] Sep 07 10:45:00.150622 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.150598 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" Sep 07 10:45:00.150891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.150865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f"] Sep 07 10:45:00.152664 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.152633 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:45:00.153376 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.153357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" Sep 07 10:45:00.154740 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.154663 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j"] Sep 07 10:45:00.157762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.157740 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" Sep 07 10:45:00.158912 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.158888 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n"] Sep 07 10:45:00.162709 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.162684 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j"] Sep 07 10:45:00.165505 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.165438 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f"] Sep 07 10:45:00.174460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.174437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" Sep 07 10:45:00.192259 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.191892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" Sep 07 10:45:00.246378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.246342 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" Sep 07 10:45:00.492170 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.492140 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n"] Sep 07 10:45:00.494125 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:45:00.494097 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d40f88_c523_4a1a_a20e_0c6127455b0d.slice/crio-4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f WatchSource:0}: Error finding container 4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f: Status 404 returned error can't find the container with id 4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f Sep 07 10:45:00.498251 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.498232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f"] Sep 07 10:45:00.500199 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:45:00.500176 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba8af53_4f8d_4db9_a625_e7c261febd1a.slice/crio-1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc WatchSource:0}: Error finding container 1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc: Status 404 returned error can't find the container with id 1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc Sep 07 10:45:00.570534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.570509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j"] Sep 07 10:45:00.572685 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:45:00.572662 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7447a2d3_ab4c_4ee5_b3db_d5d026a47cbd.slice/crio-e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3 WatchSource:0}: Error finding container e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3: Status 404 returned error can't find the container with id e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3 Sep 07 10:45:00.941213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.941178 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 07 10:45:00.941350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:00.941231 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 07 10:45:01.397686 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.397635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" event={"ID":"7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd","Type":"ContainerStarted","Data":"a4b34350e1c81dcf626d0900b6d7782c4f5ba3cecca5d620193283bf113269be"} Sep 07 10:45:01.397884 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.397703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" event={"ID":"7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd","Type":"ContainerStarted","Data":"e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3"} Sep 07 10:45:01.399309 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.399276 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" event={"ID":"bba8af53-4f8d-4db9-a625-e7c261febd1a","Type":"ContainerStarted","Data":"9166e58f4f6d825748998c79864bfcab00eb00780db23d7f5f4aaec1f9e8cbe8"} Sep 07 10:45:01.399442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.399331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" event={"ID":"bba8af53-4f8d-4db9-a625-e7c261febd1a","Type":"ContainerStarted","Data":"1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc"} Sep 07 10:45:01.401127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.401104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" event={"ID":"08d40f88-c523-4a1a-a20e-0c6127455b0d","Type":"ContainerStarted","Data":"3e9677b0611a52b0ff5bf49daf07f3fa481c87452f9cc630f261dec6e495dae0"} Sep 07 10:45:01.401263 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.401154 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" event={"ID":"08d40f88-c523-4a1a-a20e-0c6127455b0d","Type":"ContainerStarted","Data":"4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f"} Sep 07 10:45:01.432292 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.432227 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" podStartSLOduration=1.432205465 podStartE2EDuration="1.432205465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.414360292 +0000 UTC m=+1469.272049609" watchObservedRunningTime="2026-09-07 10:45:01.432205465 +0000 UTC m=+1469.289894782" Sep 07 10:45:01.433409 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.433360 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" podStartSLOduration=1.433348332 podStartE2EDuration="1.433348332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.430325698 +0000 UTC m=+1469.288015008" watchObservedRunningTime="2026-09-07 10:45:01.433348332 +0000 UTC m=+1469.291037650" Sep 07 10:45:01.446484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.446438 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" podStartSLOduration=1.446419266 podStartE2EDuration="1.446419266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.443188239 +0000 UTC m=+1469.300877554" watchObservedRunningTime="2026-09-07 10:45:01.446419266 +0000 UTC m=+1469.304108646" Sep 07 10:45:01.940922 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.940890 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 07 10:45:01.941320 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:01.940958 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 07 10:45:02.409706 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.409664 2577 generic.go:356] "Generic (PLEG): container finished" podID="08d40f88-c523-4a1a-a20e-0c6127455b0d" containerID="3e9677b0611a52b0ff5bf49daf07f3fa481c87452f9cc630f261dec6e495dae0" exitCode=0 Sep 07 10:45:02.409918 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.409791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" event={"ID":"08d40f88-c523-4a1a-a20e-0c6127455b0d","Type":"ContainerDied","Data":"3e9677b0611a52b0ff5bf49daf07f3fa481c87452f9cc630f261dec6e495dae0"} Sep 07 10:45:02.411999 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.411968 2577 generic.go:356] "Generic (PLEG): container finished" podID="7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd" containerID="a4b34350e1c81dcf626d0900b6d7782c4f5ba3cecca5d620193283bf113269be" exitCode=0 Sep 07 10:45:02.412107 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.412043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" event={"ID":"7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd","Type":"ContainerDied","Data":"a4b34350e1c81dcf626d0900b6d7782c4f5ba3cecca5d620193283bf113269be"} Sep 07 10:45:02.414245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.414218 2577 generic.go:356] "Generic (PLEG): container finished" podID="bba8af53-4f8d-4db9-a625-e7c261febd1a" containerID="9166e58f4f6d825748998c79864bfcab00eb00780db23d7f5f4aaec1f9e8cbe8" exitCode=0 Sep 07 10:45:02.414354 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.414274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" event={"ID":"bba8af53-4f8d-4db9-a625-e7c261febd1a","Type":"ContainerDied","Data":"9166e58f4f6d825748998c79864bfcab00eb00780db23d7f5f4aaec1f9e8cbe8"} Sep 07 10:45:02.940961 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.940929 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 07 10:45:02.941337 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:02.940988 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 07 10:45:03.649662 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:03.649577 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" Sep 07 10:45:03.682045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:03.682011 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" Sep 07 10:45:03.686470 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:03.686407 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" Sep 07 10:45:03.941248 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:03.941170 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 07 10:45:03.941248 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:03.941220 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 07 10:45:04.441448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.441420 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" Sep 07 10:45:04.441650 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.441448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-5lc6n" event={"ID":"08d40f88-c523-4a1a-a20e-0c6127455b0d","Type":"ContainerDied","Data":"4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f"} Sep 07 10:45:04.441650 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.441490 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4199b16f58a869b1053207648b0cc8f1e5741c23e4b00f3fb7ab63c55881a44f" Sep 07 10:45:04.442975 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.442953 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" Sep 07 10:45:04.443122 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.442954 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j" event={"ID":"7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd","Type":"ContainerDied","Data":"e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3"} Sep 07 10:45:04.443122 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.443066 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e624e8a691e8c1c30f2b6bc4cee028357e712bd29b07fdeca8628c38aa8d22e3" Sep 07 10:45:04.444485 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.444425 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" Sep 07 10:45:04.444485 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.444468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f" event={"ID":"bba8af53-4f8d-4db9-a625-e7c261febd1a","Type":"ContainerDied","Data":"1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc"} Sep 07 10:45:04.444762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.444520 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fed9648b6624d9381de2ca29dcd5b71ee34e36d4c6f1feb1a4c34e06ce03abc" Sep 07 10:45:04.944849 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.944821 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:45:04.945366 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.945342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:45:04.946032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:04.946015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:45:23.076651 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.076619 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:45:23.077262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077236 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba8af53-4f8d-4db9-a625-e7c261febd1a" containerName="cleanup" Sep 07 10:45:23.077262 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077259 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba8af53-4f8d-4db9-a625-e7c261febd1a" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077279 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077285 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077296 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d40f88-c523-4a1a-a20e-0c6127455b0d" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077302 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d40f88-c523-4a1a-a20e-0c6127455b0d" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077383 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077394 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba8af53-4f8d-4db9-a625-e7c261febd1a" containerName="cleanup" Sep 07 10:45:23.077460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.077401 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d40f88-c523-4a1a-a20e-0c6127455b0d" containerName="cleanup" Sep 07 10:45:23.082208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.082186 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.085199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.085175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2700585f\"" Sep 07 10:45:23.085301 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.085253 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2700585f-dockercfg-h7pkg\"" Sep 07 10:45:23.094324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.094294 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:45:23.098049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.098011 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:45:23.102467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.102445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.116688 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.116666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:45:23.234321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.234277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.234321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.234322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.234565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.234409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9k7\" (UniqueName: \"kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.234565 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.234441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5fxk\" (UniqueName: \"kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.335122 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9k7\" (UniqueName: \"kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.335122 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5fxk\" (UniqueName: \"kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.335356 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.335356 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.335932 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335912 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.336002 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.335913 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.342769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.342747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9k7\" (UniqueName: \"kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7\") pod \"payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.342859 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.342831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5fxk\" (UniqueName: \"kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk\") pod \"payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.396468 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.396441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:23.414354 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.414333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:23.677146 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.677119 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:45:23.678706 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:45:23.678676 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfafe5fe3_0ed6_4869_be6a_d37799cb462c.slice/crio-6a76460f5b46a2e90b21c5d4f692d66f5ca093917e2ef3a73a65f2125105e492 WatchSource:0}: Error finding container 6a76460f5b46a2e90b21c5d4f692d66f5ca093917e2ef3a73a65f2125105e492: Status 404 returned error can't find the container with id 6a76460f5b46a2e90b21c5d4f692d66f5ca093917e2ef3a73a65f2125105e492 Sep 07 10:45:23.709216 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:23.709189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:45:23.715959 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:45:23.715933 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3117d32d_7344_49c0_9d1b_f3751e88d202.slice/crio-8b8b20f116ae3b92e549666efae9d1af198363fcdd1b527186e0045e225914e5 WatchSource:0}: Error finding container 8b8b20f116ae3b92e549666efae9d1af198363fcdd1b527186e0045e225914e5: Status 404 returned error can't find the container with id 8b8b20f116ae3b92e549666efae9d1af198363fcdd1b527186e0045e225914e5 Sep 07 10:45:24.566372 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.566334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" event={"ID":"3117d32d-7344-49c0-9d1b-f3751e88d202","Type":"ContainerStarted","Data":"57fcee339768552208f14b2177e4ba27073ec9bb048641e616e0da1eb002677f"} Sep 07 10:45:24.566813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.566407 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:24.566813 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.566436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" event={"ID":"3117d32d-7344-49c0-9d1b-f3751e88d202","Type":"ContainerStarted","Data":"8b8b20f116ae3b92e549666efae9d1af198363fcdd1b527186e0045e225914e5"} Sep 07 10:45:24.567865 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.567841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" event={"ID":"fafe5fe3-0ed6-4869-be6a-d37799cb462c","Type":"ContainerStarted","Data":"0033fc1952ffec073970f9ad415e456fbed575e617f8d7f93c2aef418a23ab55"} Sep 07 10:45:24.567999 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.567895 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:45:24.567999 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.567917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" event={"ID":"fafe5fe3-0ed6-4869-be6a-d37799cb462c","Type":"ContainerStarted","Data":"6a76460f5b46a2e90b21c5d4f692d66f5ca093917e2ef3a73a65f2125105e492"} Sep 07 10:45:24.582499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.582452 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" podStartSLOduration=1.582436181 podStartE2EDuration="1.582436181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:24.582002329 +0000 UTC m=+1492.439691731" watchObservedRunningTime="2026-09-07 10:45:24.582436181 +0000 UTC m=+1492.440125494" Sep 07 10:45:24.598079 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:24.598037 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" podStartSLOduration=1.5980230579999999 podStartE2EDuration="1.598023058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:24.596711502 +0000 UTC m=+1492.454400819" watchObservedRunningTime="2026-09-07 10:45:24.598023058 +0000 UTC m=+1492.455712374" Sep 07 10:45:35.575373 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:35.575340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:45:35.576003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:45:35.575415 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:46:32.869036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:32.869005 2577 scope.go:117] "RemoveContainer" containerID="ce419d38d940b9691fe04414de3f5758bc913c90feb7b464259eaeb099bcf78d" Sep 07 10:46:32.878356 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:32.878337 2577 scope.go:117] "RemoveContainer" containerID="a19c3174479f5975257756b855501c9bc8d8881d6c9c8c822eebd74ccc33a9d0" Sep 07 10:46:42.217718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.217683 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn"] Sep 07 10:46:42.222966 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.222942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:42.225540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.225518 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:46:42.226487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.225874 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:46:42.228280 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.228253 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn"] Sep 07 10:46:42.310387 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.310350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2700585f-7qdfn\" (UID: \"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:42.411050 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.411014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2700585f-7qdfn\" (UID: \"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:42.411690 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.411666 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2700585f-7qdfn\" (UID: \"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:42.537496 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.537463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:42.817835 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:42.817811 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn"] Sep 07 10:46:42.819988 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:46:42.819965 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd51fd7_1bb2_4731_ae9a_5de16f335cf9.slice/crio-726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0 WatchSource:0}: Error finding container 726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0: Status 404 returned error can't find the container with id 726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0 Sep 07 10:46:43.048392 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:43.048362 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" event={"ID":"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9","Type":"ContainerStarted","Data":"ab4f66b8aad1be0841e67e41f5813ba07b807a346ede8c627508d11850e67f9a"} Sep 07 10:46:43.048518 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:43.048423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" event={"ID":"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9","Type":"ContainerStarted","Data":"726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0"} Sep 07 10:46:43.064782 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:43.064739 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" podStartSLOduration=1.064723955 podStartE2EDuration="1.064723955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:46:43.06156115 +0000 UTC m=+1570.919250463" watchObservedRunningTime="2026-09-07 10:46:43.064723955 +0000 UTC m=+1570.922413271" Sep 07 10:46:44.055907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:44.055872 2577 generic.go:356] "Generic (PLEG): container finished" podID="cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" containerID="ab4f66b8aad1be0841e67e41f5813ba07b807a346ede8c627508d11850e67f9a" exitCode=0 Sep 07 10:46:44.056259 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:44.055956 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" event={"ID":"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9","Type":"ContainerDied","Data":"ab4f66b8aad1be0841e67e41f5813ba07b807a346ede8c627508d11850e67f9a"} Sep 07 10:46:45.192146 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:45.192120 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:45.235124 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:45.235097 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca\") pod \"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9\" (UID: \"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9\") " Sep 07 10:46:45.235435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:45.235409 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" (UID: "cdd51fd7-1bb2-4731-ae9a-5de16f335cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:46:45.337124 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:45.337008 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:46:46.068022 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:46.067991 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" Sep 07 10:46:46.068197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:46.068028 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn" event={"ID":"cdd51fd7-1bb2-4731-ae9a-5de16f335cf9","Type":"ContainerDied","Data":"726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0"} Sep 07 10:46:46.068197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:46.068086 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="726abca27b10a75fda34b8dd4885d460647af600124ebae03a9e6ea771b8cdb0" Sep 07 10:46:57.905403 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:57.905371 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:46:57.905880 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:57.905651 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" podUID="3117d32d-7344-49c0-9d1b-f3751e88d202" containerName="payload-processing" containerID="cri-o://57fcee339768552208f14b2177e4ba27073ec9bb048641e616e0da1eb002677f" gracePeriod=30 Sep 07 10:46:57.910431 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:57.910405 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:46:57.910874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:57.910825 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" podUID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" containerName="payload-pre-processing" containerID="cri-o://0033fc1952ffec073970f9ad415e456fbed575e617f8d7f93c2aef418a23ab55" gracePeriod=30 Sep 07 10:46:58.097276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.097231 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn"] Sep 07 10:46:58.100110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.100085 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2700585f-7qdfn"] Sep 07 10:46:58.140874 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.140839 2577 generic.go:356] "Generic (PLEG): container finished" podID="3117d32d-7344-49c0-9d1b-f3751e88d202" containerID="57fcee339768552208f14b2177e4ba27073ec9bb048641e616e0da1eb002677f" exitCode=0 Sep 07 10:46:58.141021 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.140913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" event={"ID":"3117d32d-7344-49c0-9d1b-f3751e88d202","Type":"ContainerDied","Data":"57fcee339768552208f14b2177e4ba27073ec9bb048641e616e0da1eb002677f"} Sep 07 10:46:58.142899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.142877 2577 generic.go:356] "Generic (PLEG): container finished" podID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" containerID="0033fc1952ffec073970f9ad415e456fbed575e617f8d7f93c2aef418a23ab55" exitCode=0 Sep 07 10:46:58.143039 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.142957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" event={"ID":"fafe5fe3-0ed6-4869-be6a-d37799cb462c","Type":"ContainerDied","Data":"0033fc1952ffec073970f9ad415e456fbed575e617f8d7f93c2aef418a23ab55"} Sep 07 10:46:58.205678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.205648 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:46:58.208760 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.208744 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:46:58.263263 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263232 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw9k7\" (UniqueName: \"kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7\") pod \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " Sep 07 10:46:58.263444 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263316 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume\") pod \"3117d32d-7344-49c0-9d1b-f3751e88d202\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " Sep 07 10:46:58.263444 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263341 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume\") pod \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\" (UID: \"fafe5fe3-0ed6-4869-be6a-d37799cb462c\") " Sep 07 10:46:58.263444 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263385 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5fxk\" (UniqueName: \"kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk\") pod \"3117d32d-7344-49c0-9d1b-f3751e88d202\" (UID: \"3117d32d-7344-49c0-9d1b-f3751e88d202\") " Sep 07 10:46:58.263784 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263753 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3117d32d-7344-49c0-9d1b-f3751e88d202" (UID: "3117d32d-7344-49c0-9d1b-f3751e88d202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:46:58.263784 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263763 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fafe5fe3-0ed6-4869-be6a-d37799cb462c" (UID: "fafe5fe3-0ed6-4869-be6a-d37799cb462c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:46:58.264147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263871 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3117d32d-7344-49c0-9d1b-f3751e88d202-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:46:58.264147 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.263885 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fafe5fe3-0ed6-4869-be6a-d37799cb462c-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:46:58.265437 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.265418 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7" (OuterVolumeSpecName: "kube-api-access-nw9k7") pod "fafe5fe3-0ed6-4869-be6a-d37799cb462c" (UID: "fafe5fe3-0ed6-4869-be6a-d37799cb462c"). InnerVolumeSpecName "kube-api-access-nw9k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:46:58.265925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.265909 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk" (OuterVolumeSpecName: "kube-api-access-t5fxk") pod "3117d32d-7344-49c0-9d1b-f3751e88d202" (UID: "3117d32d-7344-49c0-9d1b-f3751e88d202"). InnerVolumeSpecName "kube-api-access-t5fxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:46:58.364714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.364680 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9k7\" (UniqueName: \"kubernetes.io/projected/fafe5fe3-0ed6-4869-be6a-d37799cb462c-kube-api-access-nw9k7\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:46:58.364714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.364708 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5fxk\" (UniqueName: \"kubernetes.io/projected/3117d32d-7344-49c0-9d1b-f3751e88d202-kube-api-access-t5fxk\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:46:58.723768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:58.723733 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" path="/var/lib/kubelet/pods/cdd51fd7-1bb2-4731-ae9a-5de16f335cf9/volumes" Sep 07 10:46:59.149997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.149901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" event={"ID":"3117d32d-7344-49c0-9d1b-f3751e88d202","Type":"ContainerDied","Data":"8b8b20f116ae3b92e549666efae9d1af198363fcdd1b527186e0045e225914e5"} Sep 07 10:46:59.149997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.149908 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2" Sep 07 10:46:59.150492 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.149970 2577 scope.go:117] "RemoveContainer" containerID="57fcee339768552208f14b2177e4ba27073ec9bb048641e616e0da1eb002677f" Sep 07 10:46:59.151466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.151363 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" Sep 07 10:46:59.151466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.151400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck" event={"ID":"fafe5fe3-0ed6-4869-be6a-d37799cb462c","Type":"ContainerDied","Data":"6a76460f5b46a2e90b21c5d4f692d66f5ca093917e2ef3a73a65f2125105e492"} Sep 07 10:46:59.159617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.159607 2577 scope.go:117] "RemoveContainer" containerID="0033fc1952ffec073970f9ad415e456fbed575e617f8d7f93c2aef418a23ab55" Sep 07 10:46:59.170033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.170006 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:46:59.174460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.174418 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2700585f-565dbf6594-9f7ck"] Sep 07 10:46:59.183151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.183132 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:46:59.186912 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:46:59.186893 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2700585f-545b4bd5cc-z8ln2"] Sep 07 10:47:00.723830 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:47:00.723797 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3117d32d-7344-49c0-9d1b-f3751e88d202" path="/var/lib/kubelet/pods/3117d32d-7344-49c0-9d1b-f3751e88d202/volumes" Sep 07 10:47:00.724208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:47:00.724158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" path="/var/lib/kubelet/pods/fafe5fe3-0ed6-4869-be6a-d37799cb462c/volumes" Sep 07 10:47:32.905255 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:47:32.905182 2577 scope.go:117] "RemoveContainer" containerID="4db68f8dc5672238e6acfbcdf8839126b18eb65b1b79341c1966de01619f26d5" Sep 07 10:48:32.936841 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:32.936805 2577 scope.go:117] "RemoveContainer" containerID="2f1f9d2cf7ff8fd731d7610b72e97ca6449d08cd72bc4d192ea3b8422a35add9" Sep 07 10:48:45.415615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.415561 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:48:45.416279 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416263 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" containerName="revoke-keys" Sep 07 10:48:45.416352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416283 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" containerName="revoke-keys" Sep 07 10:48:45.416352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416338 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" containerName="payload-pre-processing" Sep 07 10:48:45.416352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416347 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" containerName="payload-pre-processing" Sep 07 10:48:45.416471 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416363 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3117d32d-7344-49c0-9d1b-f3751e88d202" containerName="payload-processing" Sep 07 10:48:45.416471 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416372 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3117d32d-7344-49c0-9d1b-f3751e88d202" containerName="payload-processing" Sep 07 10:48:45.416471 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416459 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdd51fd7-1bb2-4731-ae9a-5de16f335cf9" containerName="revoke-keys" Sep 07 10:48:45.416615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416476 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fafe5fe3-0ed6-4869-be6a-d37799cb462c" containerName="payload-pre-processing" Sep 07 10:48:45.416615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.416485 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3117d32d-7344-49c0-9d1b-f3751e88d202" containerName="payload-processing" Sep 07 10:48:45.421658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.421637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.423868 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.423843 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4e2c0-openshift-default-dockercfg-lf7km\"" Sep 07 10:48:45.431667 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.431644 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:48:45.541271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541475 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541545 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541545 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6j7\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541702 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541696 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.541875 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.541722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642839 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6j7\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.642997 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.642936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643069 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643314 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643486 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643486 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643621 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643518 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.643821 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.643803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.645850 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.645828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.646045 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.646030 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.650721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.650695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.650904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.650879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6j7\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7\") pod \"e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:45.743415 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:45.743382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:46.028058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.028031 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:48:46.029451 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:48:46.029422 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085504b1_302f_45e4_9a71_d41fe7637543.slice/crio-79c6dc657cc41895d05d68a3674d4297e86fb9174beb0bdaf9d0f70c46377e3f WatchSource:0}: Error finding container 79c6dc657cc41895d05d68a3674d4297e86fb9174beb0bdaf9d0f70c46377e3f: Status 404 returned error can't find the container with id 79c6dc657cc41895d05d68a3674d4297e86fb9174beb0bdaf9d0f70c46377e3f Sep 07 10:48:46.031847 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.031816 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:48:46.031940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.031886 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:48:46.031940 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.031913 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:48:46.787871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.787830 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerStarted","Data":"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521"} Sep 07 10:48:46.788365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.787899 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerStarted","Data":"79c6dc657cc41895d05d68a3674d4297e86fb9174beb0bdaf9d0f70c46377e3f"} Sep 07 10:48:46.807938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:46.807881 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podStartSLOduration=1.807861528 podStartE2EDuration="1.807861528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:46.807005692 +0000 UTC m=+1694.664695004" watchObservedRunningTime="2026-09-07 10:48:46.807861528 +0000 UTC m=+1694.665550844" Sep 07 10:48:47.743698 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:47.743640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:47.749881 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:47.749849 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:47.793376 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:47.793339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:47.794211 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:47.794188 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:48:50.558280 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.558234 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:48:50.562242 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.562219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.564807 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.564778 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4e2c0\"" Sep 07 10:48:50.564807 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.564795 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4e2c0-dockercfg-lz4tj\"" Sep 07 10:48:50.577732 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.577688 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:48:50.700110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.700060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6f9\" (UniqueName: \"kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.700310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.700122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.801256 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.801210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6f9\" (UniqueName: \"kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.801494 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.801269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.801994 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.801974 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.809711 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.809639 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6f9\" (UniqueName: \"kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9\") pod \"payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:50.887076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:50.887038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:51.169391 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:51.169358 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:48:51.171942 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:48:51.171899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f916bd_928e_4770_aa22_9617815e1b74.slice/crio-69ca88ebad5d52bf4ffda2c84e84eecb31c27b64fd15a23f183e48e4af633846 WatchSource:0}: Error finding container 69ca88ebad5d52bf4ffda2c84e84eecb31c27b64fd15a23f183e48e4af633846: Status 404 returned error can't find the container with id 69ca88ebad5d52bf4ffda2c84e84eecb31c27b64fd15a23f183e48e4af633846 Sep 07 10:48:51.819927 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:51.819882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" event={"ID":"b5f916bd-928e-4770-aa22-9617815e1b74","Type":"ContainerStarted","Data":"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6"} Sep 07 10:48:51.820350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:51.819957 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" event={"ID":"b5f916bd-928e-4770-aa22-9617815e1b74","Type":"ContainerStarted","Data":"69ca88ebad5d52bf4ffda2c84e84eecb31c27b64fd15a23f183e48e4af633846"} Sep 07 10:48:51.820350 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:51.820078 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:48:51.838812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:48:51.838736 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" podStartSLOduration=1.838719282 podStartE2EDuration="1.838719282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:51.835427002 +0000 UTC m=+1699.693116331" watchObservedRunningTime="2026-09-07 10:48:51.838719282 +0000 UTC m=+1699.696408598" Sep 07 10:49:02.828920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:02.828812 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:49:10.953990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.953950 2577 generic.go:356] "Generic (PLEG): container finished" podID="085504b1-302f-45e4-9a71-d41fe7637543" containerID="c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521" exitCode=0 Sep 07 10:49:10.954419 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.954027 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerDied","Data":"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521"} Sep 07 10:49:10.954531 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.954516 2577 scope.go:117] "RemoveContainer" containerID="c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521" Sep 07 10:49:10.955974 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.955948 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:10.956049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.956017 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:10.956049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:10.956045 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:11.962145 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:11.962108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerStarted","Data":"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf"} Sep 07 10:49:12.743990 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:12.743956 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:49:12.744208 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:12.744190 2577 patch_prober.go:28] interesting pod/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:49:12.744256 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:12.744232 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 07 10:49:13.743964 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:13.743921 2577 patch_prober.go:28] interesting pod/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:49:13.744373 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:13.743979 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 07 10:49:14.348195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.348158 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:49:14.362398 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.362368 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.365579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.365537 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4f3dee-openshift-default-dockercfg-5z5pw\"" Sep 07 10:49:14.367259 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.367229 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:49:14.469192 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469140 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469201 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469411 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mkg\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.469722 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.469610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.570918 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.570879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.570944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mkg\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571526 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571767 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.571809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.571788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.574164 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.574137 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.574379 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.574359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.582397 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.582364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.582533 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.582465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mkg\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg\") pod \"e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.677736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.677628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:14.744574 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.744536 2577 patch_prober.go:28] interesting pod/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:49:14.744969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.744627 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 07 10:49:14.980370 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.980342 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:49:14.982359 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:49:14.982326 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce196911_27d8_4085_ab25_5d7ccd1ccc03.slice/crio-81d29ddbd18f9cd526abc4437a06b9eeceb6ae18582f4b05ca73ad41e4ce7cde WatchSource:0}: Error finding container 81d29ddbd18f9cd526abc4437a06b9eeceb6ae18582f4b05ca73ad41e4ce7cde: Status 404 returned error can't find the container with id 81d29ddbd18f9cd526abc4437a06b9eeceb6ae18582f4b05ca73ad41e4ce7cde Sep 07 10:49:14.984992 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.984953 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:14.985087 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.985049 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:14.985127 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:14.985099 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:15.744048 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:15.744008 2577 patch_prober.go:28] interesting pod/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:49:15.744328 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:15.744075 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 07 10:49:15.990217 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:15.990172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerStarted","Data":"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459"} Sep 07 10:49:15.990633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:15.990240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerStarted","Data":"81d29ddbd18f9cd526abc4437a06b9eeceb6ae18582f4b05ca73ad41e4ce7cde"} Sep 07 10:49:16.014073 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.013935 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podStartSLOduration=2.013907787 podStartE2EDuration="2.013907787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:49:16.009892634 +0000 UTC m=+1723.867581959" watchObservedRunningTime="2026-09-07 10:49:16.013907787 +0000 UTC m=+1723.871597104" Sep 07 10:49:16.678223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.678179 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:16.683945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.683915 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:16.744763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.744725 2577 patch_prober.go:28] interesting pod/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 07 10:49:16.744945 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.744789 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 07 10:49:16.996613 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.996552 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:16.997714 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:16.997694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:17.749140 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:17.749108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:49:17.749420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:17.749403 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:49:17.750159 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:17.750137 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:49:19.916613 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:19.915614 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:49:19.924519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:19.924486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:19.931226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:19.930775 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4f3dee\"" Sep 07 10:49:19.931226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:19.931038 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4f3dee-dockercfg-8hzxl\"" Sep 07 10:49:19.936489 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:19.936435 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:49:20.037098 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.037050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.037303 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.037253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lrp\" (UniqueName: \"kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.138046 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.138002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.138279 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.138192 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lrp\" (UniqueName: \"kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.139080 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.139054 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.146260 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.146230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lrp\" (UniqueName: \"kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp\") pod \"payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.246278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.246240 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:20.525799 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:20.525762 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:49:20.526868 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:49:20.526841 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb67981_99f5_4b78_a476_3984d20bb495.slice/crio-99dfcfb6fb813bfc54c609743468ee7648fdb3944e00dc05e48120ed1627aa68 WatchSource:0}: Error finding container 99dfcfb6fb813bfc54c609743468ee7648fdb3944e00dc05e48120ed1627aa68: Status 404 returned error can't find the container with id 99dfcfb6fb813bfc54c609743468ee7648fdb3944e00dc05e48120ed1627aa68 Sep 07 10:49:21.025978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:21.025933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" event={"ID":"9bb67981-99f5-4b78-a476-3984d20bb495","Type":"ContainerStarted","Data":"4277938eb8e3ccb8ca406f68bf184f4bc16801a1f5d5151fd443e36141f977a3"} Sep 07 10:49:21.026422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:21.026003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" event={"ID":"9bb67981-99f5-4b78-a476-3984d20bb495","Type":"ContainerStarted","Data":"99dfcfb6fb813bfc54c609743468ee7648fdb3944e00dc05e48120ed1627aa68"} Sep 07 10:49:21.026422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:21.026126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:21.043027 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:21.042968 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" podStartSLOduration=2.042951161 podStartE2EDuration="2.042951161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:49:21.04092788 +0000 UTC m=+1728.898617197" watchObservedRunningTime="2026-09-07 10:49:21.042951161 +0000 UTC m=+1728.900640477" Sep 07 10:49:32.035168 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:32.035124 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:49:32.966788 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:32.966756 2577 scope.go:117] "RemoveContainer" containerID="5736024955bb7f8797ac6eaa36ea6b70520a2df83b3716d65f197b633047f1b8" Sep 07 10:49:32.976917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:32.976895 2577 scope.go:117] "RemoveContainer" containerID="3f5a298da83abe8adc55f6ca381bee2bcac6b7907d9b1f5736b9791b2c6d8784" Sep 07 10:49:40.150097 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.150015 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerID="c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459" exitCode=0 Sep 07 10:49:40.150538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.150088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerDied","Data":"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459"} Sep 07 10:49:40.150538 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.150512 2577 scope.go:117] "RemoveContainer" containerID="c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459" Sep 07 10:49:40.151834 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.151796 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:40.152128 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.151876 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:40.152128 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:40.151918 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:49:41.164430 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:41.164351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerStarted","Data":"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1"} Sep 07 10:49:41.678169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:41.678134 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:41.678422 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:41.678392 2577 patch_prober.go:28] interesting pod/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 07 10:49:41.678542 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:41.678454 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 07 10:49:42.678244 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:42.678208 2577 patch_prober.go:28] interesting pod/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 07 10:49:42.678692 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:42.678273 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 07 10:49:43.678637 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:43.678570 2577 patch_prober.go:28] interesting pod/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 07 10:49:43.679024 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:43.678657 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 07 10:49:44.678643 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:44.678609 2577 patch_prober.go:28] interesting pod/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 07 10:49:44.679049 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:44.678678 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 07 10:49:45.678097 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:45.678057 2577 patch_prober.go:28] interesting pod/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 07 10:49:45.678293 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:45.678136 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 07 10:49:46.683951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:46.683907 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:46.684431 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:46.684220 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:49:46.685058 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:49:46.685040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:50:02.704113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:02.704071 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:50:02.704734 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:02.704402 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" containerName="payload-processing" containerID="cri-o://862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6" gracePeriod=30 Sep 07 10:50:02.826462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:02.826425 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.76:9004: connect: connection refused" Sep 07 10:50:02.974729 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:02.974701 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:50:03.041605 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.040891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj6f9\" (UniqueName: \"kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9\") pod \"b5f916bd-928e-4770-aa22-9617815e1b74\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " Sep 07 10:50:03.041605 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.041099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume\") pod \"b5f916bd-928e-4770-aa22-9617815e1b74\" (UID: \"b5f916bd-928e-4770-aa22-9617815e1b74\") " Sep 07 10:50:03.049620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.048049 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5f916bd-928e-4770-aa22-9617815e1b74" (UID: "b5f916bd-928e-4770-aa22-9617815e1b74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:03.064773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.064429 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9" (OuterVolumeSpecName: "kube-api-access-wj6f9") pod "b5f916bd-928e-4770-aa22-9617815e1b74" (UID: "b5f916bd-928e-4770-aa22-9617815e1b74"). InnerVolumeSpecName "kube-api-access-wj6f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:03.149271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.149216 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5f916bd-928e-4770-aa22-9617815e1b74-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:03.149271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.149264 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj6f9\" (UniqueName: \"kubernetes.io/projected/b5f916bd-928e-4770-aa22-9617815e1b74-kube-api-access-wj6f9\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:03.312575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.312540 2577 generic.go:356] "Generic (PLEG): container finished" podID="b5f916bd-928e-4770-aa22-9617815e1b74" containerID="862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6" exitCode=0 Sep 07 10:50:03.312810 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.312614 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" Sep 07 10:50:03.312810 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.312652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" event={"ID":"b5f916bd-928e-4770-aa22-9617815e1b74","Type":"ContainerDied","Data":"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6"} Sep 07 10:50:03.312810 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.312717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8" event={"ID":"b5f916bd-928e-4770-aa22-9617815e1b74","Type":"ContainerDied","Data":"69ca88ebad5d52bf4ffda2c84e84eecb31c27b64fd15a23f183e48e4af633846"} Sep 07 10:50:03.312810 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.312746 2577 scope.go:117] "RemoveContainer" containerID="862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6" Sep 07 10:50:03.322488 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.322472 2577 scope.go:117] "RemoveContainer" containerID="862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6" Sep 07 10:50:03.322776 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:03.322758 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6\": container with ID starting with 862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6 not found: ID does not exist" containerID="862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6" Sep 07 10:50:03.322838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.322784 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6"} err="failed to get container status \"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6\": rpc error: code = NotFound desc = could not find container \"862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6\": container with ID starting with 862f9e7a658f23bb23b65741c5d7ffedfbf6cfd3cc7efbfb6c93a3ec986222e6 not found: ID does not exist" Sep 07 10:50:03.334483 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.334445 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:50:03.337799 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:03.337772 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4e2c0-6996794c7f-j5mw8"] Sep 07 10:50:04.724512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:04.724480 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" path="/var/lib/kubelet/pods/b5f916bd-928e-4770-aa22-9617815e1b74/volumes" Sep 07 10:50:09.308910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:09.308869 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:50:09.309297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:09.309141 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" containerID="cri-o://f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf" gracePeriod=30 Sep 07 10:50:14.571288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.571225 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:50:14.677642 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677573 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.677897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677654 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.677897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.677897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677728 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.677897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677837 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.677897 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677878 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.678199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.678199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:14.678199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677976 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:14.678199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.677971 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6j7\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.678199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo\") pod \"085504b1-302f-45e4-9a71-d41fe7637543\" (UID: \"085504b1-302f-45e4-9a71-d41fe7637543\") " Sep 07 10:50:14.678466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:14.678466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678226 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:14.678466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678212 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data" (OuterVolumeSpecName: "istio-data") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:14.678840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678613 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.678840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678633 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.678840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678643 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.678840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678653 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.678840 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.678661 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/085504b1-302f-45e4-9a71-d41fe7637543-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.680324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.680295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7" (OuterVolumeSpecName: "kube-api-access-fh6j7") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "kube-api-access-fh6j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:14.680449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.680319 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:50:14.680577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.680553 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token" (OuterVolumeSpecName: "istio-token") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:14.680692 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.680634 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "085504b1-302f-45e4-9a71-d41fe7637543" (UID: "085504b1-302f-45e4-9a71-d41fe7637543"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:14.779294 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.779251 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/085504b1-302f-45e4-9a71-d41fe7637543-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.779294 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.779288 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6j7\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-kube-api-access-fh6j7\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.779294 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.779301 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/085504b1-302f-45e4-9a71-d41fe7637543-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:14.779541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:14.779315 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/085504b1-302f-45e4-9a71-d41fe7637543-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:15.390822 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.390782 2577 generic.go:356] "Generic (PLEG): container finished" podID="085504b1-302f-45e4-9a71-d41fe7637543" containerID="f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf" exitCode=0 Sep 07 10:50:15.391005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.390847 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" Sep 07 10:50:15.391005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.390855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerDied","Data":"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf"} Sep 07 10:50:15.391005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.390922 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9" event={"ID":"085504b1-302f-45e4-9a71-d41fe7637543","Type":"ContainerDied","Data":"79c6dc657cc41895d05d68a3674d4297e86fb9174beb0bdaf9d0f70c46377e3f"} Sep 07 10:50:15.391005 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.390948 2577 scope.go:117] "RemoveContainer" containerID="f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf" Sep 07 10:50:15.401096 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.401077 2577 scope.go:117] "RemoveContainer" containerID="c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521" Sep 07 10:50:15.408498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.408473 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:50:15.411237 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.411217 2577 scope.go:117] "RemoveContainer" containerID="f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf" Sep 07 10:50:15.411570 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:15.411548 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf\": container with ID starting with f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf not found: ID does not exist" containerID="f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf" Sep 07 10:50:15.411658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.411578 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf"} err="failed to get container status \"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf\": rpc error: code = NotFound desc = could not find container \"f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf\": container with ID starting with f5d14d6b42a0829b6ff9ce0d2fd3ff534a11037bdbbf981fee81a87e897799cf not found: ID does not exist" Sep 07 10:50:15.411658 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.411619 2577 scope.go:117] "RemoveContainer" containerID="c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521" Sep 07 10:50:15.411900 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:15.411884 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521\": container with ID starting with c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521 not found: ID does not exist" containerID="c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521" Sep 07 10:50:15.411936 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.411908 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521"} err="failed to get container status \"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521\": rpc error: code = NotFound desc = could not find container \"c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521\": container with ID starting with c4a7d7b0be4721e7319d3b6f36686cd35a3275061edc9fd572d5b46698f3e521 not found: ID does not exist" Sep 07 10:50:15.412395 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:15.412378 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4e2c0-openshift-default-75f4cbc596-q9cc9"] Sep 07 10:50:16.727115 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:16.727075 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="085504b1-302f-45e4-9a71-d41fe7637543" path="/var/lib/kubelet/pods/085504b1-302f-45e4-9a71-d41fe7637543/volumes" Sep 07 10:50:25.240195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.239982 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:50:25.240640 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.240245 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" podUID="9bb67981-99f5-4b78-a476-3984d20bb495" containerName="payload-processing" containerID="cri-o://4277938eb8e3ccb8ca406f68bf184f4bc16801a1f5d5151fd443e36141f977a3" gracePeriod=30 Sep 07 10:50:25.455371 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.455338 2577 generic.go:356] "Generic (PLEG): container finished" podID="9bb67981-99f5-4b78-a476-3984d20bb495" containerID="4277938eb8e3ccb8ca406f68bf184f4bc16801a1f5d5151fd443e36141f977a3" exitCode=0 Sep 07 10:50:25.455547 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.455418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" event={"ID":"9bb67981-99f5-4b78-a476-3984d20bb495","Type":"ContainerDied","Data":"4277938eb8e3ccb8ca406f68bf184f4bc16801a1f5d5151fd443e36141f977a3"} Sep 07 10:50:25.511786 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.511756 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:50:25.702444 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.702407 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7lrp\" (UniqueName: \"kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp\") pod \"9bb67981-99f5-4b78-a476-3984d20bb495\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " Sep 07 10:50:25.702649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.702510 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume\") pod \"9bb67981-99f5-4b78-a476-3984d20bb495\" (UID: \"9bb67981-99f5-4b78-a476-3984d20bb495\") " Sep 07 10:50:25.702891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.702858 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bb67981-99f5-4b78-a476-3984d20bb495" (UID: "9bb67981-99f5-4b78-a476-3984d20bb495"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:25.703056 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.703043 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb67981-99f5-4b78-a476-3984d20bb495-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:25.704783 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.704759 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp" (OuterVolumeSpecName: "kube-api-access-f7lrp") pod "9bb67981-99f5-4b78-a476-3984d20bb495" (UID: "9bb67981-99f5-4b78-a476-3984d20bb495"). InnerVolumeSpecName "kube-api-access-f7lrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:25.811338 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:25.811247 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lrp\" (UniqueName: \"kubernetes.io/projected/9bb67981-99f5-4b78-a476-3984d20bb495-kube-api-access-f7lrp\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:26.462663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.462615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" event={"ID":"9bb67981-99f5-4b78-a476-3984d20bb495","Type":"ContainerDied","Data":"99dfcfb6fb813bfc54c609743468ee7648fdb3944e00dc05e48120ed1627aa68"} Sep 07 10:50:26.463100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.462671 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj" Sep 07 10:50:26.463100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.462720 2577 scope.go:117] "RemoveContainer" containerID="4277938eb8e3ccb8ca406f68bf184f4bc16801a1f5d5151fd443e36141f977a3" Sep 07 10:50:26.483240 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.483213 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:50:26.491621 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.491595 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4f3dee-5584fb89f9-lvwhj"] Sep 07 10:50:26.723841 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:26.723801 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb67981-99f5-4b78-a476-3984d20bb495" path="/var/lib/kubelet/pods/9bb67981-99f5-4b78-a476-3984d20bb495/volumes" Sep 07 10:50:31.888649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:31.888615 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:50:31.889110 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:31.888892 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" containerID="cri-o://103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1" gracePeriod=30 Sep 07 10:50:32.253855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.253811 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c"] Sep 07 10:50:32.254297 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" containerName="payload-processing" Sep 07 10:50:32.254342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254298 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" containerName="payload-processing" Sep 07 10:50:32.254342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254306 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb67981-99f5-4b78-a476-3984d20bb495" containerName="payload-processing" Sep 07 10:50:32.254342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254311 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb67981-99f5-4b78-a476-3984d20bb495" containerName="payload-processing" Sep 07 10:50:32.254342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254326 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254342 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254332 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254355 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254361 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254444 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb67981-99f5-4b78-a476-3984d20bb495" containerName="payload-processing" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254454 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254471 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="085504b1-302f-45e4-9a71-d41fe7637543" containerName="istio-proxy" Sep 07 10:50:32.254527 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.254479 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5f916bd-928e-4770-aa22-9617815e1b74" containerName="payload-processing" Sep 07 10:50:32.257812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.257793 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:32.259875 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.259843 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:50:32.260030 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.259880 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:50:32.267194 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.267165 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c"] Sep 07 10:50:32.269559 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.269529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c\" (UID: \"05503ccc-a55c-4ad4-b299-f77334d99bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:32.371013 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.370978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c\" (UID: \"05503ccc-a55c-4ad4-b299-f77334d99bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:32.371646 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.371625 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c\" (UID: \"05503ccc-a55c-4ad4-b299-f77334d99bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:32.575876 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.575789 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:50:32.583019 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.582997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:32.865623 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:32.865573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c"] Sep 07 10:50:32.866743 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:50:32.866713 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05503ccc_a55c_4ad4_b299_f77334d99bbb.slice/crio-dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8 WatchSource:0}: Error finding container dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8: Status 404 returned error can't find the container with id dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8 Sep 07 10:50:33.007793 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:33.007761 2577 scope.go:117] "RemoveContainer" containerID="124e510c038ff2f0e54f9e0e3c33acb7af21c99b6943fae1bbd998f745b2170e" Sep 07 10:50:33.506067 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:33.506032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" event={"ID":"05503ccc-a55c-4ad4-b299-f77334d99bbb","Type":"ContainerStarted","Data":"7cedda4f93f932966c1d1c4673506674802c0e83f314acd428aac114aa5dc60a"} Sep 07 10:50:33.506257 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:33.506087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" event={"ID":"05503ccc-a55c-4ad4-b299-f77334d99bbb","Type":"ContainerStarted","Data":"dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8"} Sep 07 10:50:33.520781 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:33.520730 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" podStartSLOduration=1.520714435 podStartE2EDuration="1.520714435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:50:33.518960096 +0000 UTC m=+1801.376649437" watchObservedRunningTime="2026-09-07 10:50:33.520714435 +0000 UTC m=+1801.378403788" Sep 07 10:50:34.513633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:34.513570 2577 generic.go:356] "Generic (PLEG): container finished" podID="05503ccc-a55c-4ad4-b299-f77334d99bbb" containerID="7cedda4f93f932966c1d1c4673506674802c0e83f314acd428aac114aa5dc60a" exitCode=0 Sep 07 10:50:34.514103 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:34.513654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" event={"ID":"05503ccc-a55c-4ad4-b299-f77334d99bbb","Type":"ContainerDied","Data":"7cedda4f93f932966c1d1c4673506674802c0e83f314acd428aac114aa5dc60a"} Sep 07 10:50:35.662947 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:35.662919 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:35.705033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:35.704976 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca\") pod \"05503ccc-a55c-4ad4-b299-f77334d99bbb\" (UID: \"05503ccc-a55c-4ad4-b299-f77334d99bbb\") " Sep 07 10:50:35.705476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:35.705446 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05503ccc-a55c-4ad4-b299-f77334d99bbb" (UID: "05503ccc-a55c-4ad4-b299-f77334d99bbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:35.805921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:35.805817 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05503ccc-a55c-4ad4-b299-f77334d99bbb-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:36.526797 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:36.526763 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" Sep 07 10:50:36.527003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:36.526814 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c" event={"ID":"05503ccc-a55c-4ad4-b299-f77334d99bbb","Type":"ContainerDied","Data":"dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8"} Sep 07 10:50:36.527003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:36.526872 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfede4b576c565d1ceeb65bc4a25402a0ef22b9b5b264f2af666500ca1ef4bb8" Sep 07 10:50:37.149391 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.149367 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:50:37.221291 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221258 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221344 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7mkg\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221368 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221398 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221474 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221508 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221553 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:37.221851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221574 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data\") pod \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\" (UID: \"ce196911-27d8-4085-ab25-5d7ccd1ccc03\") " Sep 07 10:50:37.221851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221839 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:37.222060 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:37.222060 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.221956 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:37.222169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.222060 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:37.222169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.222144 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.222169 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.222164 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.222492 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.222178 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.223976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.223941 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg" (OuterVolumeSpecName: "kube-api-access-v7mkg") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "kube-api-access-v7mkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:37.223976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.223948 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:50:37.223976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.223954 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:37.223976 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.223936 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce196911-27d8-4085-ab25-5d7ccd1ccc03" (UID: "ce196911-27d8-4085-ab25-5d7ccd1ccc03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:50:37.323496 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323455 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.323496 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323493 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mkg\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-kube-api-access-v7mkg\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.323757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323506 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.323757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323519 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.323757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323530 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.323757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.323540 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce196911-27d8-4085-ab25-5d7ccd1ccc03-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:37.534546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.534435 2577 generic.go:356] "Generic (PLEG): container finished" podID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerID="103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1" exitCode=0 Sep 07 10:50:37.534546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.534489 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" Sep 07 10:50:37.534546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.534528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerDied","Data":"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1"} Sep 07 10:50:37.534871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.534617 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk" event={"ID":"ce196911-27d8-4085-ab25-5d7ccd1ccc03","Type":"ContainerDied","Data":"81d29ddbd18f9cd526abc4437a06b9eeceb6ae18582f4b05ca73ad41e4ce7cde"} Sep 07 10:50:37.534871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.534654 2577 scope.go:117] "RemoveContainer" containerID="103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1" Sep 07 10:50:37.546653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.546572 2577 scope.go:117] "RemoveContainer" containerID="c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459" Sep 07 10:50:37.556179 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.556153 2577 scope.go:117] "RemoveContainer" containerID="103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1" Sep 07 10:50:37.556514 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:37.556494 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1\": container with ID starting with 103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1 not found: ID does not exist" containerID="103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1" Sep 07 10:50:37.556628 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.556527 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1"} err="failed to get container status \"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1\": rpc error: code = NotFound desc = could not find container \"103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1\": container with ID starting with 103e383e7dbbb2db1377463070bbcf6d1058715477b43148da4ab11cbfa569a1 not found: ID does not exist" Sep 07 10:50:37.556628 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.556549 2577 scope.go:117] "RemoveContainer" containerID="c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459" Sep 07 10:50:37.556839 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:37.556818 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459\": container with ID starting with c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459 not found: ID does not exist" containerID="c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459" Sep 07 10:50:37.556904 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.556844 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459"} err="failed to get container status \"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459\": rpc error: code = NotFound desc = could not find container \"c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459\": container with ID starting with c3b056c3fb150ede88770c97f31ca7447f2ef94d933d61f16863dee761955459 not found: ID does not exist" Sep 07 10:50:37.558573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.558548 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:50:37.562315 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:37.562292 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4f3dee-openshift-default-7b55dd9d6d-h6tjk"] Sep 07 10:50:38.723924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:38.723894 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" path="/var/lib/kubelet/pods/ce196911-27d8-4085-ab25-5d7ccd1ccc03/volumes" Sep 07 10:50:42.771693 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:42.771639 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:50:42.772164 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:42.771927 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" podUID="5a3ff872-6409-46cd-990f-68a60632fdbe" containerName="payload-processing" containerID="cri-o://209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944" gracePeriod=30 Sep 07 10:50:42.959924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:42.959880 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c"] Sep 07 10:50:42.962243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:42.962215 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a158df-wvk5c"] Sep 07 10:50:43.037052 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.036988 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:50:43.082968 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.082942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzpfv\" (UniqueName: \"kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv\") pod \"5a3ff872-6409-46cd-990f-68a60632fdbe\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " Sep 07 10:50:43.083108 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.083039 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume\") pod \"5a3ff872-6409-46cd-990f-68a60632fdbe\" (UID: \"5a3ff872-6409-46cd-990f-68a60632fdbe\") " Sep 07 10:50:43.083420 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.083401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a3ff872-6409-46cd-990f-68a60632fdbe" (UID: "5a3ff872-6409-46cd-990f-68a60632fdbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:43.085148 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.085124 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv" (OuterVolumeSpecName: "kube-api-access-rzpfv") pod "5a3ff872-6409-46cd-990f-68a60632fdbe" (UID: "5a3ff872-6409-46cd-990f-68a60632fdbe"). InnerVolumeSpecName "kube-api-access-rzpfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:50:43.183943 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.183914 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzpfv\" (UniqueName: \"kubernetes.io/projected/5a3ff872-6409-46cd-990f-68a60632fdbe-kube-api-access-rzpfv\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:43.183943 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.183945 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a3ff872-6409-46cd-990f-68a60632fdbe-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:43.569611 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.569556 2577 generic.go:356] "Generic (PLEG): container finished" podID="5a3ff872-6409-46cd-990f-68a60632fdbe" containerID="209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944" exitCode=0 Sep 07 10:50:43.569792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.569617 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" Sep 07 10:50:43.569792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.569620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" event={"ID":"5a3ff872-6409-46cd-990f-68a60632fdbe","Type":"ContainerDied","Data":"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944"} Sep 07 10:50:43.569792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.569679 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f" event={"ID":"5a3ff872-6409-46cd-990f-68a60632fdbe","Type":"ContainerDied","Data":"4371c1216acc0dfed20d8d59dd487a7f144ee7b249f3c4036f29eb86f8174e26"} Sep 07 10:50:43.569792 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.569701 2577 scope.go:117] "RemoveContainer" containerID="209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944" Sep 07 10:50:43.579192 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.579175 2577 scope.go:117] "RemoveContainer" containerID="209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944" Sep 07 10:50:43.579459 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:50:43.579443 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944\": container with ID starting with 209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944 not found: ID does not exist" containerID="209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944" Sep 07 10:50:43.579512 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.579468 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944"} err="failed to get container status \"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944\": rpc error: code = NotFound desc = could not find container \"209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944\": container with ID starting with 209105fef0718a179706e949b650f2829b75c69859161b90c3d7a3fa5d5a0944 not found: ID does not exist" Sep 07 10:50:43.595427 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.595404 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:50:43.601550 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:43.601528 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-a158df-9ccd46fc9-xn85f"] Sep 07 10:50:44.723511 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:44.723475 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05503ccc-a55c-4ad4-b299-f77334d99bbb" path="/var/lib/kubelet/pods/05503ccc-a55c-4ad4-b299-f77334d99bbb/volumes" Sep 07 10:50:44.723921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:44.723882 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3ff872-6409-46cd-990f-68a60632fdbe" path="/var/lib/kubelet/pods/5a3ff872-6409-46cd-990f-68a60632fdbe/volumes" Sep 07 10:50:49.750282 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750237 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2"] Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750758 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750791 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750796 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750805 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3ff872-6409-46cd-990f-68a60632fdbe" containerName="payload-processing" Sep 07 10:50:49.750812 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750810 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3ff872-6409-46cd-990f-68a60632fdbe" containerName="payload-processing" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750824 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05503ccc-a55c-4ad4-b299-f77334d99bbb" containerName="revoke-keys" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750830 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="05503ccc-a55c-4ad4-b299-f77334d99bbb" containerName="revoke-keys" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750909 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3ff872-6409-46cd-990f-68a60632fdbe" containerName="payload-processing" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750922 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750932 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce196911-27d8-4085-ab25-5d7ccd1ccc03" containerName="istio-proxy" Sep 07 10:50:49.751014 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.750938 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="05503ccc-a55c-4ad4-b299-f77334d99bbb" containerName="revoke-keys" Sep 07 10:50:49.755485 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.755469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:49.758083 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.758065 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:50:49.759146 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.759133 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:50:49.772626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.772597 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2"] Sep 07 10:50:49.853163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.853130 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2\" (UID: \"56eeaa50-c8da-4449-8138-c28fc86dfe59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:49.954015 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.953982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2\" (UID: \"56eeaa50-c8da-4449-8138-c28fc86dfe59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:49.954600 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:49.954549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2\" (UID: \"56eeaa50-c8da-4449-8138-c28fc86dfe59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:50.065974 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:50.065898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:50.337284 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:50.337264 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2"] Sep 07 10:50:50.339906 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:50:50.339870 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56eeaa50_c8da_4449_8138_c28fc86dfe59.slice/crio-37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df WatchSource:0}: Error finding container 37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df: Status 404 returned error can't find the container with id 37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df Sep 07 10:50:50.609926 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:50.609889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" event={"ID":"56eeaa50-c8da-4449-8138-c28fc86dfe59","Type":"ContainerStarted","Data":"37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df"} Sep 07 10:50:51.617081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:51.617042 2577 generic.go:356] "Generic (PLEG): container finished" podID="56eeaa50-c8da-4449-8138-c28fc86dfe59" containerID="1ce77dedc464913966ab563afa5a626f86f67102b7c96e2e5c05d60c93ed5d43" exitCode=0 Sep 07 10:50:51.617467 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:51.617128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" event={"ID":"56eeaa50-c8da-4449-8138-c28fc86dfe59","Type":"ContainerDied","Data":"1ce77dedc464913966ab563afa5a626f86f67102b7c96e2e5c05d60c93ed5d43"} Sep 07 10:50:52.752397 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:52.752372 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:52.883464 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:52.883393 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca\") pod \"56eeaa50-c8da-4449-8138-c28fc86dfe59\" (UID: \"56eeaa50-c8da-4449-8138-c28fc86dfe59\") " Sep 07 10:50:52.883731 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:52.883709 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56eeaa50-c8da-4449-8138-c28fc86dfe59" (UID: "56eeaa50-c8da-4449-8138-c28fc86dfe59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:50:52.984491 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:52.984466 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56eeaa50-c8da-4449-8138-c28fc86dfe59-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:50:53.629603 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:53.629554 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" event={"ID":"56eeaa50-c8da-4449-8138-c28fc86dfe59","Type":"ContainerDied","Data":"37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df"} Sep 07 10:50:53.629750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:53.629604 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2" Sep 07 10:50:53.629750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:50:53.629617 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f52be01934268fcbd412531ba4b9f171abc4d53486bf974616f2bb5b0417df" Sep 07 10:51:00.347928 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.347894 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f"] Sep 07 10:51:00.350144 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.350115 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5e934f-29812965-gsk9f"] Sep 07 10:51:00.382196 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.382166 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:51:00.382557 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.382520 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" podUID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" containerName="payload-pre-processing" containerID="cri-o://a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8" gracePeriod=30 Sep 07 10:51:00.543151 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.543119 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2"] Sep 07 10:51:00.545215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.545194 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5e934f-7lbw2"] Sep 07 10:51:00.637185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.637163 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:51:00.670070 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.670033 2577 generic.go:356] "Generic (PLEG): container finished" podID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" containerID="a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8" exitCode=0 Sep 07 10:51:00.670226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.670124 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" Sep 07 10:51:00.670226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.670153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" event={"ID":"a0b16f0b-f8e3-4660-9284-cb40f4dc461b","Type":"ContainerDied","Data":"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8"} Sep 07 10:51:00.670226 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.670210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv" event={"ID":"a0b16f0b-f8e3-4660-9284-cb40f4dc461b","Type":"ContainerDied","Data":"fc78a3ddfc6f147c1b721bce2beba5e7a3c2a41b754469658df0f649398e6ab6"} Sep 07 10:51:00.670358 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.670240 2577 scope.go:117] "RemoveContainer" containerID="a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8" Sep 07 10:51:00.679475 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.679460 2577 scope.go:117] "RemoveContainer" containerID="a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8" Sep 07 10:51:00.679756 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:51:00.679739 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8\": container with ID starting with a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8 not found: ID does not exist" containerID="a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8" Sep 07 10:51:00.679808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.679763 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8"} err="failed to get container status \"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8\": rpc error: code = NotFound desc = could not find container \"a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8\": container with ID starting with a9576f98e8f5f3f8d617e8423700733ce537807986c4f5a51505d3f860e22de8 not found: ID does not exist" Sep 07 10:51:00.730031 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.729990 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56eeaa50-c8da-4449-8138-c28fc86dfe59" path="/var/lib/kubelet/pods/56eeaa50-c8da-4449-8138-c28fc86dfe59/volumes" Sep 07 10:51:00.730455 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.730389 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba8af53-4f8d-4db9-a625-e7c261febd1a" path="/var/lib/kubelet/pods/bba8af53-4f8d-4db9-a625-e7c261febd1a/volumes" Sep 07 10:51:00.756969 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.756945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume\") pod \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " Sep 07 10:51:00.757125 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.756977 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdxth\" (UniqueName: \"kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth\") pod \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\" (UID: \"a0b16f0b-f8e3-4660-9284-cb40f4dc461b\") " Sep 07 10:51:00.757382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.757339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0b16f0b-f8e3-4660-9284-cb40f4dc461b" (UID: "a0b16f0b-f8e3-4660-9284-cb40f4dc461b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:51:00.757509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.757429 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:00.759271 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.759243 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth" (OuterVolumeSpecName: "kube-api-access-fdxth") pod "a0b16f0b-f8e3-4660-9284-cb40f4dc461b" (UID: "a0b16f0b-f8e3-4660-9284-cb40f4dc461b"). InnerVolumeSpecName "kube-api-access-fdxth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:51:00.858672 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.858642 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdxth\" (UniqueName: \"kubernetes.io/projected/a0b16f0b-f8e3-4660-9284-cb40f4dc461b-kube-api-access-fdxth\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:00.990853 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.990826 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:51:00.994638 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:00.994614 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5e934f-7c78cd6498-cqskv"] Sep 07 10:51:02.724477 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:02.724441 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" path="/var/lib/kubelet/pods/a0b16f0b-f8e3-4660-9284-cb40f4dc461b/volumes" Sep 07 10:51:07.091167 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:07.091131 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:51:07.092202 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:07.091845 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" containerID="cri-o://6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14" gracePeriod=30 Sep 07 10:51:12.346466 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.346441 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:51:12.360924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.360862 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.360924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.360895 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361095 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.360929 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361095 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.360969 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361086 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjsln\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:51:12.361207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361187 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361221 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361238 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361235 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data" (OuterVolumeSpecName: "istio-data") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:51:12.361385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361294 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert\") pod \"74011a9d-e056-4bed-8b3e-bc1703cfca72\" (UID: \"74011a9d-e056-4bed-8b3e-bc1703cfca72\") " Sep 07 10:51:12.361385 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361365 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:51:12.361685 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361482 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:51:12.361763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361731 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:51:12.361763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361742 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.361763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361760 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.361924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361773 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.361924 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.361787 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.363536 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.363507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:51:12.363705 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.363686 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token" (OuterVolumeSpecName: "istio-token") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:51:12.363779 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.363691 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln" (OuterVolumeSpecName: "kube-api-access-bjsln") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "kube-api-access-bjsln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:51:12.363878 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.363833 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74011a9d-e056-4bed-8b3e-bc1703cfca72" (UID: "74011a9d-e056-4bed-8b3e-bc1703cfca72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:51:12.462430 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.462404 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.462430 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.462430 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjsln\" (UniqueName: \"kubernetes.io/projected/74011a9d-e056-4bed-8b3e-bc1703cfca72-kube-api-access-bjsln\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.462612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.462439 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.462612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.462448 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74011a9d-e056-4bed-8b3e-bc1703cfca72-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.462612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.462457 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74011a9d-e056-4bed-8b3e-bc1703cfca72-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:51:12.742790 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.742760 2577 generic.go:356] "Generic (PLEG): container finished" podID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerID="6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14" exitCode=0 Sep 07 10:51:12.742911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.742806 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" Sep 07 10:51:12.742911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.742843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerDied","Data":"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14"} Sep 07 10:51:12.742911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.742894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95" event={"ID":"74011a9d-e056-4bed-8b3e-bc1703cfca72","Type":"ContainerDied","Data":"9f7bcc158033c2fe764bbfd464c23fd43bbbee681bb168485b43d408f5dcf4ca"} Sep 07 10:51:12.743012 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.742917 2577 scope.go:117] "RemoveContainer" containerID="6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14" Sep 07 10:51:12.752265 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.752246 2577 scope.go:117] "RemoveContainer" containerID="1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3" Sep 07 10:51:12.760137 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.760117 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:51:12.761361 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.761338 2577 scope.go:117] "RemoveContainer" containerID="6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14" Sep 07 10:51:12.761703 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:51:12.761669 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14\": container with ID starting with 6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14 not found: ID does not exist" containerID="6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14" Sep 07 10:51:12.761814 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.761705 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14"} err="failed to get container status \"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14\": rpc error: code = NotFound desc = could not find container \"6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14\": container with ID starting with 6fc248cd4fb69d7942bcb7f42e0d31f84587536efdd79213542257f239b7ad14 not found: ID does not exist" Sep 07 10:51:12.761814 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.761728 2577 scope.go:117] "RemoveContainer" containerID="1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3" Sep 07 10:51:12.763042 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:51:12.763019 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3\": container with ID starting with 1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3 not found: ID does not exist" containerID="1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3" Sep 07 10:51:12.763139 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.763052 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3"} err="failed to get container status \"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3\": rpc error: code = NotFound desc = could not find container \"1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3\": container with ID starting with 1ef6378ad7e0a33d1b9735d9d067ad3211ab11165b87284371f501e9c474edc3 not found: ID does not exist" Sep 07 10:51:12.764245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:12.764226 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5e934f-openshift-default-55cdf6d7b8-dlk95"] Sep 07 10:51:14.723860 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:14.723828 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" path="/var/lib/kubelet/pods/74011a9d-e056-4bed-8b3e-bc1703cfca72/volumes" Sep 07 10:51:26.014296 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014261 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:51:26.014753 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014739 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014755 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014764 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014769 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014788 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" containerName="payload-pre-processing" Sep 07 10:51:26.014796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014794 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" containerName="payload-pre-processing" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014804 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56eeaa50-c8da-4449-8138-c28fc86dfe59" containerName="revoke-keys" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014813 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="56eeaa50-c8da-4449-8138-c28fc86dfe59" containerName="revoke-keys" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014880 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014891 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0b16f0b-f8e3-4660-9284-cb40f4dc461b" containerName="payload-pre-processing" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014898 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74011a9d-e056-4bed-8b3e-bc1703cfca72" containerName="istio-proxy" Sep 07 10:51:26.014952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.014905 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="56eeaa50-c8da-4449-8138-c28fc86dfe59" containerName="revoke-keys" Sep 07 10:51:26.019637 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.019617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.021989 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.021968 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bef02f-openshift-default-dockercfg-gf4r6\"" Sep 07 10:51:26.028756 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.028735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:51:26.087984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.087956 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088099 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.087995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088099 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088019 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb5rk\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088181 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088181 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088309 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.088465 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.088428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.189930 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.189891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.189942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.189971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.189994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb5rk\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190116 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190150 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190338 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190415 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190642 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190474 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190747 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.190956 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.190939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.192528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.192506 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.192785 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.192764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.197913 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.197889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.198541 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.198519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb5rk\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk\") pod \"e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.330952 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.330883 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:26.606386 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.606362 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:51:26.608546 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:51:26.608513 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod710528d9_1e2c_433d_ad68_b3c78a3bf660.slice/crio-010bf0a7a4b630e7d8d5e8c409d63236505c171c62941ab2bd95e805c72e1442 WatchSource:0}: Error finding container 010bf0a7a4b630e7d8d5e8c409d63236505c171c62941ab2bd95e805c72e1442: Status 404 returned error can't find the container with id 010bf0a7a4b630e7d8d5e8c409d63236505c171c62941ab2bd95e805c72e1442 Sep 07 10:51:26.610907 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.610874 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:26.610988 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.610938 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:26.610988 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.610963 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:26.821708 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.821672 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerStarted","Data":"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb"} Sep 07 10:51:26.821909 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.821756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerStarted","Data":"010bf0a7a4b630e7d8d5e8c409d63236505c171c62941ab2bd95e805c72e1442"} Sep 07 10:51:26.841418 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:26.841371 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podStartSLOduration=1.841357645 podStartE2EDuration="1.841357645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:51:26.83900872 +0000 UTC m=+1854.696698042" watchObservedRunningTime="2026-09-07 10:51:26.841357645 +0000 UTC m=+1854.699046961" Sep 07 10:51:27.331382 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:27.331350 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:27.336612 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:27.336576 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:27.827666 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:27.827634 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:27.828655 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:27.828636 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:30.611832 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.611798 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:51:30.615451 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.615434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.617446 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.617414 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bef02f\"" Sep 07 10:51:30.617556 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.617452 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bef02f-dockercfg-wrb2l\"" Sep 07 10:51:30.625937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.625909 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:51:30.740254 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.740214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.740425 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.740278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnx7\" (UniqueName: \"kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.841513 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.841477 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.841744 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.841530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnx7\" (UniqueName: \"kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.842275 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.842250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.849674 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.849643 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnx7\" (UniqueName: \"kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7\") pod \"payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:30.927751 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:30.927655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:31.201575 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:31.201548 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:51:31.203914 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:51:31.203884 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4e160f_767d_431f_8b0b_9efcca397bac.slice/crio-b991d09a33ec28b9a8117503156a51cf586bf308c52c5c2b3bdc4b43882fa677 WatchSource:0}: Error finding container b991d09a33ec28b9a8117503156a51cf586bf308c52c5c2b3bdc4b43882fa677: Status 404 returned error can't find the container with id b991d09a33ec28b9a8117503156a51cf586bf308c52c5c2b3bdc4b43882fa677 Sep 07 10:51:31.854003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:31.853959 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" event={"ID":"5d4e160f-767d-431f-8b0b-9efcca397bac","Type":"ContainerStarted","Data":"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a"} Sep 07 10:51:31.854463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:31.854077 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" event={"ID":"5d4e160f-767d-431f-8b0b-9efcca397bac","Type":"ContainerStarted","Data":"b991d09a33ec28b9a8117503156a51cf586bf308c52c5c2b3bdc4b43882fa677"} Sep 07 10:51:31.854661 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:31.854637 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:31.873847 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:31.873745 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" podStartSLOduration=1.873711509 podStartE2EDuration="1.873711509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:51:31.871264568 +0000 UTC m=+1859.728953884" watchObservedRunningTime="2026-09-07 10:51:31.873711509 +0000 UTC m=+1859.731400819" Sep 07 10:51:33.043378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:33.043345 2577 scope.go:117] "RemoveContainer" containerID="9166e58f4f6d825748998c79864bfcab00eb00780db23d7f5f4aaec1f9e8cbe8" Sep 07 10:51:42.861632 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:42.861574 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:51:43.869728 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:43.869690 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 07 10:51:43.870160 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:43.869827 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 07 10:51:45.945082 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.945041 2577 generic.go:356] "Generic (PLEG): container finished" podID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerID="03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb" exitCode=0 Sep 07 10:51:45.945615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.945111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerDied","Data":"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb"} Sep 07 10:51:45.945615 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.945563 2577 scope.go:117] "RemoveContainer" containerID="03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb" Sep 07 10:51:45.946759 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.946730 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:45.946831 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.946816 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:45.946872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:45.946853 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:51:46.954635 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:46.954568 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerStarted","Data":"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea"} Sep 07 10:51:47.331365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:47.331325 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:47.331670 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:47.331645 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:51:47.331785 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:47.331707 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 07 10:51:48.332298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:48.332250 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:51:48.332856 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:48.332329 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 07 10:51:49.331543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:49.331505 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:51:49.331757 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:49.331668 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 07 10:51:50.331521 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:50.331481 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:51:50.332118 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:50.331546 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 07 10:51:51.331863 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:51.331812 2577 patch_prober.go:28] interesting pod/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Sep 07 10:51:51.332401 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:51.331902 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Sep 07 10:51:52.463086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:52.463047 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:52.464094 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:52.464064 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:52.464857 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:52.464838 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:51:59.897943 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.897849 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:51:59.901185 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.901142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:51:59.903387 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.903357 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-738923\"" Sep 07 10:51:59.913891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.913865 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:51:59.963894 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.963854 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhc8\" (UniqueName: \"kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:51:59.964081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:51:59.964016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.067100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.067057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhc8\" (UniqueName: \"kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.067463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.067444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.068696 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.068670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.087981 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.087950 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhc8\" (UniqueName: \"kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8\") pod \"payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.213809 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.213719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:00.488232 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:00.488199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:52:00.489782 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:52:00.489745 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95cc37e_eddd_49b2_a167_c8219aa64d88.slice/crio-51e4968b316f5a4ebc4a5388a3ce1915e2f041e9a9f832a74f65cafc9c9cd7b3 WatchSource:0}: Error finding container 51e4968b316f5a4ebc4a5388a3ce1915e2f041e9a9f832a74f65cafc9c9cd7b3: Status 404 returned error can't find the container with id 51e4968b316f5a4ebc4a5388a3ce1915e2f041e9a9f832a74f65cafc9c9cd7b3 Sep 07 10:52:01.065975 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:01.065939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" event={"ID":"f95cc37e-eddd-49b2-a167-c8219aa64d88","Type":"ContainerStarted","Data":"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495"} Sep 07 10:52:01.066534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:01.065994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" event={"ID":"f95cc37e-eddd-49b2-a167-c8219aa64d88","Type":"ContainerStarted","Data":"51e4968b316f5a4ebc4a5388a3ce1915e2f041e9a9f832a74f65cafc9c9cd7b3"} Sep 07 10:52:01.066534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:01.066089 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:01.081570 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:01.081139 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" podStartSLOduration=2.081112147 podStartE2EDuration="2.081112147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:52:01.080549543 +0000 UTC m=+1888.938238861" watchObservedRunningTime="2026-09-07 10:52:01.081112147 +0000 UTC m=+1888.938801464" Sep 07 10:52:12.074487 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:12.074452 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:52:28.114246 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:52:28.114214 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 10:53:05.907415 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:05.907375 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn_5d4e160f-767d-431f-8b0b-9efcca397bac/payload-processing/0.log" Sep 07 10:53:06.034514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.034487 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 10:53:06.687197 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.687161 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:06.691053 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.691032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.692920 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.692901 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-39c953-openshift-default-dockercfg-kp9mh\"" Sep 07 10:53:06.707322 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.704271 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:06.768194 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768300 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznsk\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768359 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768487 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768570 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.768621 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.768605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.869987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznsk\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870412 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870576 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870415 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.870576 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.870454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.871198 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.871159 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.871470 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.871444 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.871719 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.871659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.871999 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.871967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.872250 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.872220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.875178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.874190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.877699 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.877673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.880396 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.880364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:06.881465 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:06.881422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznsk\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk\") pod \"e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:07.008310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.008287 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:07.287464 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.287439 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:07.289797 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:53:07.289762 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04723e0_4525_4ce0_97ef_bd08c6e20c07.slice/crio-7d811b1176c5d6a866c67c4de80bf8df93406e1eccabd26a88872947d5f4db19 WatchSource:0}: Error finding container 7d811b1176c5d6a866c67c4de80bf8df93406e1eccabd26a88872947d5f4db19: Status 404 returned error can't find the container with id 7d811b1176c5d6a866c67c4de80bf8df93406e1eccabd26a88872947d5f4db19 Sep 07 10:53:07.292100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.292064 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:07.292178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.292134 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:07.292178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.292162 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:07.443228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.443181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerStarted","Data":"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5"} Sep 07 10:53:07.443426 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.443251 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerStarted","Data":"7d811b1176c5d6a866c67c4de80bf8df93406e1eccabd26a88872947d5f4db19"} Sep 07 10:53:07.462191 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:07.462133 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podStartSLOduration=1.462113616 podStartE2EDuration="1.462113616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:53:07.460729677 +0000 UTC m=+1955.318419011" watchObservedRunningTime="2026-09-07 10:53:07.462113616 +0000 UTC m=+1955.319802931" Sep 07 10:53:08.008755 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:08.008707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:08.014762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:08.014733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:08.449573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:08.449482 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:08.450539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:08.450520 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:11.168023 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.167984 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:11.172761 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.172733 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.175011 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.174986 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-39c953\"" Sep 07 10:53:11.175258 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.175240 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-39c953-dockercfg-trrt4\"" Sep 07 10:53:11.185680 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.185650 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:11.323821 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.323769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjgq\" (UniqueName: \"kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.323821 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.323826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.425173 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.425061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjgq\" (UniqueName: \"kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.425173 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.425114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.426042 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.426010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.435179 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.435151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjgq\" (UniqueName: \"kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq\") pod \"payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.485669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.485627 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:11.758956 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:11.758932 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:11.760833 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:53:11.760796 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac406ec_67bc_47a8_9a73_b8e77b5f169d.slice/crio-cad68a210396704e4bbbcb669fc1c5034535af994bfdb8ebe76c9237b880bc06 WatchSource:0}: Error finding container cad68a210396704e4bbbcb669fc1c5034535af994bfdb8ebe76c9237b880bc06: Status 404 returned error can't find the container with id cad68a210396704e4bbbcb669fc1c5034535af994bfdb8ebe76c9237b880bc06 Sep 07 10:53:12.474152 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:12.474114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" event={"ID":"cac406ec-67bc-47a8-9a73-b8e77b5f169d","Type":"ContainerStarted","Data":"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620"} Sep 07 10:53:12.474673 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:12.474194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" event={"ID":"cac406ec-67bc-47a8-9a73-b8e77b5f169d","Type":"ContainerStarted","Data":"cad68a210396704e4bbbcb669fc1c5034535af994bfdb8ebe76c9237b880bc06"} Sep 07 10:53:12.474673 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:12.474237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:12.491847 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:12.491795 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" podStartSLOduration=1.491777994 podStartE2EDuration="1.491777994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:53:12.489762942 +0000 UTC m=+1960.347452278" watchObservedRunningTime="2026-09-07 10:53:12.491777994 +0000 UTC m=+1960.349467311" Sep 07 10:53:23.482193 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:23.482103 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:30.595389 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.595350 2577 generic.go:356] "Generic (PLEG): container finished" podID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerID="cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5" exitCode=0 Sep 07 10:53:30.595851 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.595426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerDied","Data":"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5"} Sep 07 10:53:30.595899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.595885 2577 scope.go:117] "RemoveContainer" containerID="cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5" Sep 07 10:53:30.597021 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.596990 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:30.597138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.597076 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:30.597138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:30.597129 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 07 10:53:31.603408 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:31.603371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerStarted","Data":"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60"} Sep 07 10:53:32.009669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:32.009637 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:32.009948 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:32.009703 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 07 10:53:32.010117 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:32.010095 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 07 10:53:33.008891 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:33.008847 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 07 10:53:33.009389 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:33.008923 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 07 10:53:33.088543 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:33.088506 2577 scope.go:117] "RemoveContainer" containerID="ab4f66b8aad1be0841e67e41f5813ba07b807a346ede8c627508d11850e67f9a" Sep 07 10:53:34.009118 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:34.009074 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 07 10:53:34.009645 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:34.009151 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 07 10:53:35.009264 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:35.009213 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 07 10:53:35.009838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:35.009288 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 07 10:53:36.009727 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:36.009689 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Sep 07 10:53:36.010750 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:36.010329 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Sep 07 10:53:37.015539 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:37.015500 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:37.015998 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:37.015840 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:37.016573 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:37.016556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:41.092010 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.091926 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:41.093176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.093137 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" podUID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" containerName="payload-processing" containerID="cri-o://cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620" gracePeriod=30 Sep 07 10:53:41.347626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.347534 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:41.453872 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.453837 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kjgq\" (UniqueName: \"kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq\") pod \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " Sep 07 10:53:41.454113 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.453891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume\") pod \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\" (UID: \"cac406ec-67bc-47a8-9a73-b8e77b5f169d\") " Sep 07 10:53:41.454270 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.454242 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cac406ec-67bc-47a8-9a73-b8e77b5f169d" (UID: "cac406ec-67bc-47a8-9a73-b8e77b5f169d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:53:41.456233 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.456204 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq" (OuterVolumeSpecName: "kube-api-access-2kjgq") pod "cac406ec-67bc-47a8-9a73-b8e77b5f169d" (UID: "cac406ec-67bc-47a8-9a73-b8e77b5f169d"). InnerVolumeSpecName "kube-api-access-2kjgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:53:41.555625 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.555561 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac406ec-67bc-47a8-9a73-b8e77b5f169d-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:41.555625 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.555624 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjgq\" (UniqueName: \"kubernetes.io/projected/cac406ec-67bc-47a8-9a73-b8e77b5f169d-kube-api-access-2kjgq\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:41.674699 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.674617 2577 generic.go:356] "Generic (PLEG): container finished" podID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" containerID="cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620" exitCode=0 Sep 07 10:53:41.674838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.674708 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" Sep 07 10:53:41.674838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.674719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" event={"ID":"cac406ec-67bc-47a8-9a73-b8e77b5f169d","Type":"ContainerDied","Data":"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620"} Sep 07 10:53:41.674838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.674760 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v" event={"ID":"cac406ec-67bc-47a8-9a73-b8e77b5f169d","Type":"ContainerDied","Data":"cad68a210396704e4bbbcb669fc1c5034535af994bfdb8ebe76c9237b880bc06"} Sep 07 10:53:41.674838 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.674781 2577 scope.go:117] "RemoveContainer" containerID="cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620" Sep 07 10:53:41.684579 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.684556 2577 scope.go:117] "RemoveContainer" containerID="cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620" Sep 07 10:53:41.684881 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:53:41.684862 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620\": container with ID starting with cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620 not found: ID does not exist" containerID="cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620" Sep 07 10:53:41.684938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.684890 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620"} err="failed to get container status \"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620\": rpc error: code = NotFound desc = could not find container \"cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620\": container with ID starting with cd0c40ad34e9051cec2d81d547982c6ad4f242e711762fcc078d381ee1a3b620 not found: ID does not exist" Sep 07 10:53:41.697178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.697155 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:41.700892 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:41.700870 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-39c953-9f56f7bd7-ck55v"] Sep 07 10:53:42.723176 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:42.723144 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" path="/var/lib/kubelet/pods/cac406ec-67bc-47a8-9a73-b8e77b5f169d/volumes" Sep 07 10:53:48.198946 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:48.198908 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:48.199453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:48.199146 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" containerID="cri-o://49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60" gracePeriod=30 Sep 07 10:53:52.018449 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.018413 2577 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 07 10:53:52.018871 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.018479 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 07 10:53:52.209056 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.209018 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k"] Sep 07 10:53:52.209611 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.209573 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" containerName="payload-processing" Sep 07 10:53:52.209721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.209614 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" containerName="payload-processing" Sep 07 10:53:52.209780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.209767 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac406ec-67bc-47a8-9a73-b8e77b5f169d" containerName="payload-processing" Sep 07 10:53:52.214509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.214491 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:52.216609 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.216573 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:53:52.216609 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.216578 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:53:52.223174 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.223151 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k"] Sep 07 10:53:52.366237 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.366146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k\" (UID: \"c1b03503-5f2f-4297-9545-f8382267d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:52.466868 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.466828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k\" (UID: \"c1b03503-5f2f-4297-9545-f8382267d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:52.467476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.467458 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k\" (UID: \"c1b03503-5f2f-4297-9545-f8382267d191\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:52.527666 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.527640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:52.829509 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:52.829459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k"] Sep 07 10:53:53.438003 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.437977 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:53.577091 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577061 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577219 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qznsk\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577219 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577219 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577196 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577219 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577213 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577423 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577227 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577423 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577265 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577423 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577292 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577423 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577325 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert\") pod \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\" (UID: \"d04723e0-4525-4ce0-97ef-bd08c6e20c07\") " Sep 07 10:53:53.577669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:53:53.577669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577512 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data" (OuterVolumeSpecName: "istio-data") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:53:53.577669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577500 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:53:53.577669 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:53:53.577867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577810 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.577867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577833 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.577867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577843 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:53:53.577867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577849 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.577867 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.577864 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.606020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.605996 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:53:53.606020 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.606015 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token" (OuterVolumeSpecName: "istio-token") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:53:53.606205 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.606017 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:53:53.606205 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.606038 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk" (OuterVolumeSpecName: "kube-api-access-qznsk") pod "d04723e0-4525-4ce0-97ef-bd08c6e20c07" (UID: "d04723e0-4525-4ce0-97ef-bd08c6e20c07"). InnerVolumeSpecName "kube-api-access-qznsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:53:53.679104 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.679077 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.679104 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.679101 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.679243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.679114 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.679243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.679125 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.679243 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.679136 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qznsk\" (UniqueName: \"kubernetes.io/projected/d04723e0-4525-4ce0-97ef-bd08c6e20c07-kube-api-access-qznsk\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:53.743027 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.743001 2577 generic.go:356] "Generic (PLEG): container finished" podID="c1b03503-5f2f-4297-9545-f8382267d191" containerID="bbe2a3bcb29b5e9a7dd52a8af168ea4231cb10d15248489f7ab19cf1b6a53f70" exitCode=0 Sep 07 10:53:53.743165 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.743092 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" event={"ID":"c1b03503-5f2f-4297-9545-f8382267d191","Type":"ContainerDied","Data":"bbe2a3bcb29b5e9a7dd52a8af168ea4231cb10d15248489f7ab19cf1b6a53f70"} Sep 07 10:53:53.743165 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.743149 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" event={"ID":"c1b03503-5f2f-4297-9545-f8382267d191","Type":"ContainerStarted","Data":"4360652d5f00cd2b768df89df27365337bcb94b8d169b0507f0116236b2a5225"} Sep 07 10:53:53.744762 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.744742 2577 generic.go:356] "Generic (PLEG): container finished" podID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerID="49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60" exitCode=0 Sep 07 10:53:53.744846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.744783 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" Sep 07 10:53:53.744846 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.744820 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerDied","Data":"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60"} Sep 07 10:53:53.744921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.744870 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj" event={"ID":"d04723e0-4525-4ce0-97ef-bd08c6e20c07","Type":"ContainerDied","Data":"7d811b1176c5d6a866c67c4de80bf8df93406e1eccabd26a88872947d5f4db19"} Sep 07 10:53:53.744921 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.744899 2577 scope.go:117] "RemoveContainer" containerID="49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60" Sep 07 10:53:53.754717 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.754702 2577 scope.go:117] "RemoveContainer" containerID="cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5" Sep 07 10:53:53.763723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.763705 2577 scope.go:117] "RemoveContainer" containerID="49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60" Sep 07 10:53:53.764000 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:53:53.763980 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60\": container with ID starting with 49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60 not found: ID does not exist" containerID="49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60" Sep 07 10:53:53.764051 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.764009 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60"} err="failed to get container status \"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60\": rpc error: code = NotFound desc = could not find container \"49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60\": container with ID starting with 49d516d3b4c85e55512808eef684c2720ff454f684dd652a162ed5f39539cc60 not found: ID does not exist" Sep 07 10:53:53.764051 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.764025 2577 scope.go:117] "RemoveContainer" containerID="cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5" Sep 07 10:53:53.764270 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:53:53.764252 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5\": container with ID starting with cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5 not found: ID does not exist" containerID="cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5" Sep 07 10:53:53.764311 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.764276 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5"} err="failed to get container status \"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5\": rpc error: code = NotFound desc = could not find container \"cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5\": container with ID starting with cb89cd3bea76bda24b357ed749cdc04e0943b85d00e115e4be56ad6d2ae17ce5 not found: ID does not exist" Sep 07 10:53:53.771478 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.771449 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:53.776742 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:53.776723 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39c953-openshift-default-6d5964bdd8-ptfmj"] Sep 07 10:53:54.723721 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:54.723691 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" path="/var/lib/kubelet/pods/d04723e0-4525-4ce0-97ef-bd08c6e20c07/volumes" Sep 07 10:53:54.883206 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:54.883179 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:54.890985 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:54.890968 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca\") pod \"c1b03503-5f2f-4297-9545-f8382267d191\" (UID: \"c1b03503-5f2f-4297-9545-f8382267d191\") " Sep 07 10:53:54.891276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:54.891253 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1b03503-5f2f-4297-9545-f8382267d191" (UID: "c1b03503-5f2f-4297-9545-f8382267d191"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:53:54.991843 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:54.991768 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1b03503-5f2f-4297-9545-f8382267d191-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:53:55.759311 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:55.759283 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" Sep 07 10:53:55.759741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:55.759313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k" event={"ID":"c1b03503-5f2f-4297-9545-f8382267d191","Type":"ContainerDied","Data":"4360652d5f00cd2b768df89df27365337bcb94b8d169b0507f0116236b2a5225"} Sep 07 10:53:55.759741 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:53:55.759362 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4360652d5f00cd2b768df89df27365337bcb94b8d169b0507f0116236b2a5225" Sep 07 10:54:07.841513 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:07.841473 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:54:07.842006 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:07.841770 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" podUID="5d4e160f-767d-431f-8b0b-9efcca397bac" containerName="payload-processing" containerID="cri-o://8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a" gracePeriod=30 Sep 07 10:54:08.024718 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.024683 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k"] Sep 07 10:54:08.025352 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.025321 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bef02f-fxv6k"] Sep 07 10:54:08.102906 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.102851 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:54:08.218178 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.218148 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume\") pod \"5d4e160f-767d-431f-8b0b-9efcca397bac\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " Sep 07 10:54:08.218346 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.218250 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhnx7\" (UniqueName: \"kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7\") pod \"5d4e160f-767d-431f-8b0b-9efcca397bac\" (UID: \"5d4e160f-767d-431f-8b0b-9efcca397bac\") " Sep 07 10:54:08.218570 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.218542 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d4e160f-767d-431f-8b0b-9efcca397bac" (UID: "5d4e160f-767d-431f-8b0b-9efcca397bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:54:08.220458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.220419 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7" (OuterVolumeSpecName: "kube-api-access-xhnx7") pod "5d4e160f-767d-431f-8b0b-9efcca397bac" (UID: "5d4e160f-767d-431f-8b0b-9efcca397bac"). InnerVolumeSpecName "kube-api-access-xhnx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:54:08.319888 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.319857 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d4e160f-767d-431f-8b0b-9efcca397bac-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:54:08.319888 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.319889 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhnx7\" (UniqueName: \"kubernetes.io/projected/5d4e160f-767d-431f-8b0b-9efcca397bac-kube-api-access-xhnx7\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:54:08.723923 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.723880 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b03503-5f2f-4297-9545-f8382267d191" path="/var/lib/kubelet/pods/c1b03503-5f2f-4297-9545-f8382267d191/volumes" Sep 07 10:54:08.833443 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.833414 2577 generic.go:356] "Generic (PLEG): container finished" podID="5d4e160f-767d-431f-8b0b-9efcca397bac" containerID="8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a" exitCode=0 Sep 07 10:54:08.833577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.833461 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" Sep 07 10:54:08.833577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.833498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" event={"ID":"5d4e160f-767d-431f-8b0b-9efcca397bac","Type":"ContainerDied","Data":"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a"} Sep 07 10:54:08.833577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.833553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn" event={"ID":"5d4e160f-767d-431f-8b0b-9efcca397bac","Type":"ContainerDied","Data":"b991d09a33ec28b9a8117503156a51cf586bf308c52c5c2b3bdc4b43882fa677"} Sep 07 10:54:08.833577 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.833599 2577 scope.go:117] "RemoveContainer" containerID="8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a" Sep 07 10:54:08.844103 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.843930 2577 scope.go:117] "RemoveContainer" containerID="8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a" Sep 07 10:54:08.844336 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:54:08.844207 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a\": container with ID starting with 8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a not found: ID does not exist" containerID="8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a" Sep 07 10:54:08.844336 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.844238 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a"} err="failed to get container status \"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a\": rpc error: code = NotFound desc = could not find container \"8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a\": container with ID starting with 8a382e466e68f762c4a8965168b0b5a21533311a4fa7950697e7d858af454c3a not found: ID does not exist" Sep 07 10:54:08.853730 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.853709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:54:08.860214 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:08.860191 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bef02f-85b96c59-5xdgn"] Sep 07 10:54:10.724138 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:10.724101 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4e160f-767d-431f-8b0b-9efcca397bac" path="/var/lib/kubelet/pods/5d4e160f-767d-431f-8b0b-9efcca397bac/volumes" Sep 07 10:54:56.626012 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:56.625926 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:54:56.626439 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:56.626181 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" containerID="cri-o://5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea" gracePeriod=30 Sep 07 10:54:57.049010 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.048967 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr"] Sep 07 10:54:57.049435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049420 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:54:57.049435 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049435 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049456 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b03503-5f2f-4297-9545-f8382267d191" containerName="revoke-keys" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049462 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b03503-5f2f-4297-9545-f8382267d191" containerName="revoke-keys" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049470 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4e160f-767d-431f-8b0b-9efcca397bac" containerName="payload-processing" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049475 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4e160f-767d-431f-8b0b-9efcca397bac" containerName="payload-processing" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049481 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049487 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049545 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4e160f-767d-431f-8b0b-9efcca397bac" containerName="payload-processing" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049552 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:54:57.049572 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.049559 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b03503-5f2f-4297-9545-f8382267d191" containerName="revoke-keys" Sep 07 10:54:57.052898 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.052877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:54:57.054845 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.054827 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:54:57.054951 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.054855 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:54:57.058910 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.058882 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr"] Sep 07 10:54:57.177636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.177575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-738923-99slr\" (UID: \"5f6efef0-a0bd-4fd3-abd9-9716086fdf02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:54:57.278403 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.278366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-738923-99slr\" (UID: \"5f6efef0-a0bd-4fd3-abd9-9716086fdf02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:54:57.278972 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.278953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-738923-99slr\" (UID: \"5f6efef0-a0bd-4fd3-abd9-9716086fdf02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:54:57.366175 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.366081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:54:57.640337 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:57.639964 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr"] Sep 07 10:54:57.642935 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:54:57.642902 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6efef0_a0bd_4fd3_abd9_9716086fdf02.slice/crio-e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2 WatchSource:0}: Error finding container e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2: Status 404 returned error can't find the container with id e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2 Sep 07 10:54:58.108338 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:58.108306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" event={"ID":"5f6efef0-a0bd-4fd3-abd9-9716086fdf02","Type":"ContainerStarted","Data":"142ec07e011e6f99f329392763ed62431f62670a8303290de4823beafa2765eb"} Sep 07 10:54:58.108480 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:58.108355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" event={"ID":"5f6efef0-a0bd-4fd3-abd9-9716086fdf02","Type":"ContainerStarted","Data":"e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2"} Sep 07 10:54:58.123177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:58.123131 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" podStartSLOduration=1.123104091 podStartE2EDuration="1.123104091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:54:58.121099294 +0000 UTC m=+2065.978788612" watchObservedRunningTime="2026-09-07 10:54:58.123104091 +0000 UTC m=+2065.980793406" Sep 07 10:54:59.121723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:59.121687 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f6efef0-a0bd-4fd3-abd9-9716086fdf02" containerID="142ec07e011e6f99f329392763ed62431f62670a8303290de4823beafa2765eb" exitCode=0 Sep 07 10:54:59.122100 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:54:59.121780 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" event={"ID":"5f6efef0-a0bd-4fd3-abd9-9716086fdf02","Type":"ContainerDied","Data":"142ec07e011e6f99f329392763ed62431f62670a8303290de4823beafa2765eb"} Sep 07 10:55:00.267546 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:00.267520 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:55:00.409566 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:00.409471 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca\") pod \"5f6efef0-a0bd-4fd3-abd9-9716086fdf02\" (UID: \"5f6efef0-a0bd-4fd3-abd9-9716086fdf02\") " Sep 07 10:55:00.409915 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:00.409889 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f6efef0-a0bd-4fd3-abd9-9716086fdf02" (UID: "5f6efef0-a0bd-4fd3-abd9-9716086fdf02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:00.510659 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:00.510623 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f6efef0-a0bd-4fd3-abd9-9716086fdf02-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:01.135144 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:01.135051 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" Sep 07 10:55:01.135144 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:01.135076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr" event={"ID":"5f6efef0-a0bd-4fd3-abd9-9716086fdf02","Type":"ContainerDied","Data":"e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2"} Sep 07 10:55:01.135144 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:01.135122 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2550a1522800c02aeecb8a683a82efd5d405736dcc7e503d7fe5168b52cf7f2" Sep 07 10:55:01.873375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:01.873352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:55:02.026857 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.026821 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.026880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.026910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.026943 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027036 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.026994 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb5rk\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027073 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027085 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027104 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027142 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027173 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket\") pod \"710528d9-1e2c-433d-ad68-b3c78a3bf660\" (UID: \"710528d9-1e2c-433d-ad68-b3c78a3bf660\") " Sep 07 10:55:02.027245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027200 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:02.027532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027406 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data" (OuterVolumeSpecName: "istio-data") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:02.027532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027434 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:02.027532 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027519 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:02.027723 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027704 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710528d9-1e2c-433d-ad68-b3c78a3bf660-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.027773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027729 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.027773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027744 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.027773 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027758 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.027938 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.027793 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.029126 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.029101 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk" (OuterVolumeSpecName: "kube-api-access-vb5rk") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "kube-api-access-vb5rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:02.029330 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.029287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:55:02.029562 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.029534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:02.029687 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.029572 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token" (OuterVolumeSpecName: "istio-token") pod "710528d9-1e2c-433d-ad68-b3c78a3bf660" (UID: "710528d9-1e2c-433d-ad68-b3c78a3bf660"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:02.128386 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.128361 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb5rk\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-kube-api-access-vb5rk\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.128386 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.128383 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.128534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.128393 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.128534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.128401 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710528d9-1e2c-433d-ad68-b3c78a3bf660-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:02.142633 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.142606 2577 generic.go:356] "Generic (PLEG): container finished" podID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerID="5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea" exitCode=0 Sep 07 10:55:02.142731 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.142641 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" Sep 07 10:55:02.142731 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.142666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerDied","Data":"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea"} Sep 07 10:55:02.142731 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.142720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt" event={"ID":"710528d9-1e2c-433d-ad68-b3c78a3bf660","Type":"ContainerDied","Data":"010bf0a7a4b630e7d8d5e8c409d63236505c171c62941ab2bd95e805c72e1442"} Sep 07 10:55:02.142839 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.142745 2577 scope.go:117] "RemoveContainer" containerID="5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea" Sep 07 10:55:02.155033 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.155017 2577 scope.go:117] "RemoveContainer" containerID="03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb" Sep 07 10:55:02.163274 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.163200 2577 scope.go:117] "RemoveContainer" containerID="5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea" Sep 07 10:55:02.163698 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:02.163661 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea\": container with ID starting with 5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea not found: ID does not exist" containerID="5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea" Sep 07 10:55:02.163806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.163707 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea"} err="failed to get container status \"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea\": rpc error: code = NotFound desc = could not find container \"5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea\": container with ID starting with 5382e805e71659551eeabb218329a9aa187ced9bb95bf3e4fb3d08ef27aa6fea not found: ID does not exist" Sep 07 10:55:02.163806 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.163730 2577 scope.go:117] "RemoveContainer" containerID="03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb" Sep 07 10:55:02.164028 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:02.164009 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb\": container with ID starting with 03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb not found: ID does not exist" containerID="03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb" Sep 07 10:55:02.164089 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.164034 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb"} err="failed to get container status \"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb\": rpc error: code = NotFound desc = could not find container \"03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb\": container with ID starting with 03056433f0caeca555200117185103a74814e67c82063b88df0f32d0a000dfcb not found: ID does not exist" Sep 07 10:55:02.165848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.165826 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:55:02.173381 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.173361 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bef02f-openshift-default-d589b6fdd-6rznt"] Sep 07 10:55:02.723655 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:02.723629 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" path="/var/lib/kubelet/pods/710528d9-1e2c-433d-ad68-b3c78a3bf660/volumes" Sep 07 10:55:07.560551 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.560514 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:55:07.561081 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.560801 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" podUID="f95cc37e-eddd-49b2-a167-c8219aa64d88" containerName="payload-pre-processing" containerID="cri-o://30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495" gracePeriod=30 Sep 07 10:55:07.734068 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.734031 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr"] Sep 07 10:55:07.737430 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.737411 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-738923-99slr"] Sep 07 10:55:07.822520 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.822462 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:55:07.885796 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.885765 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqhc8\" (UniqueName: \"kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8\") pod \"f95cc37e-eddd-49b2-a167-c8219aa64d88\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " Sep 07 10:55:07.886154 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.885827 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume\") pod \"f95cc37e-eddd-49b2-a167-c8219aa64d88\" (UID: \"f95cc37e-eddd-49b2-a167-c8219aa64d88\") " Sep 07 10:55:07.886229 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.886195 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f95cc37e-eddd-49b2-a167-c8219aa64d88" (UID: "f95cc37e-eddd-49b2-a167-c8219aa64d88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:07.888195 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.888166 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8" (OuterVolumeSpecName: "kube-api-access-rqhc8") pod "f95cc37e-eddd-49b2-a167-c8219aa64d88" (UID: "f95cc37e-eddd-49b2-a167-c8219aa64d88"). InnerVolumeSpecName "kube-api-access-rqhc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:07.986684 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.986652 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95cc37e-eddd-49b2-a167-c8219aa64d88-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:07.986684 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:07.986676 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqhc8\" (UniqueName: \"kubernetes.io/projected/f95cc37e-eddd-49b2-a167-c8219aa64d88-kube-api-access-rqhc8\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:08.183421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.183333 2577 generic.go:356] "Generic (PLEG): container finished" podID="f95cc37e-eddd-49b2-a167-c8219aa64d88" containerID="30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495" exitCode=0 Sep 07 10:55:08.183421 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.183380 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" Sep 07 10:55:08.183629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.183421 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" event={"ID":"f95cc37e-eddd-49b2-a167-c8219aa64d88","Type":"ContainerDied","Data":"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495"} Sep 07 10:55:08.183629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.183474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5" event={"ID":"f95cc37e-eddd-49b2-a167-c8219aa64d88","Type":"ContainerDied","Data":"51e4968b316f5a4ebc4a5388a3ce1915e2f041e9a9f832a74f65cafc9c9cd7b3"} Sep 07 10:55:08.183629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.183498 2577 scope.go:117] "RemoveContainer" containerID="30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495" Sep 07 10:55:08.192925 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.192903 2577 scope.go:117] "RemoveContainer" containerID="30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495" Sep 07 10:55:08.193206 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:08.193181 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495\": container with ID starting with 30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495 not found: ID does not exist" containerID="30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495" Sep 07 10:55:08.193295 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.193224 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495"} err="failed to get container status \"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495\": rpc error: code = NotFound desc = could not find container \"30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495\": container with ID starting with 30cdbcbcddd52e6cfbc401564999149ef024c111b7cf3ef013a83df9d8389495 not found: ID does not exist" Sep 07 10:55:08.204369 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.204347 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:55:08.208225 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.208204 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-738923-8467b5854b-24vj5"] Sep 07 10:55:08.723856 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.723822 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6efef0-a0bd-4fd3-abd9-9716086fdf02" path="/var/lib/kubelet/pods/5f6efef0-a0bd-4fd3-abd9-9716086fdf02/volumes" Sep 07 10:55:08.724225 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:08.724175 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f95cc37e-eddd-49b2-a167-c8219aa64d88" path="/var/lib/kubelet/pods/f95cc37e-eddd-49b2-a167-c8219aa64d88/volumes" Sep 07 10:55:14.804064 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804030 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6"] Sep 07 10:55:14.804715 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804693 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f95cc37e-eddd-49b2-a167-c8219aa64d88" containerName="payload-pre-processing" Sep 07 10:55:14.804763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804718 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f95cc37e-eddd-49b2-a167-c8219aa64d88" containerName="payload-pre-processing" Sep 07 10:55:14.804763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804736 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:55:14.804763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804744 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:55:14.804855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804764 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6efef0-a0bd-4fd3-abd9-9716086fdf02" containerName="revoke-keys" Sep 07 10:55:14.804855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804773 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6efef0-a0bd-4fd3-abd9-9716086fdf02" containerName="revoke-keys" Sep 07 10:55:14.804855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804802 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:55:14.804855 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804810 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:55:14.804978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804900 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04723e0-4525-4ce0-97ef-bd08c6e20c07" containerName="istio-proxy" Sep 07 10:55:14.804978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804915 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f95cc37e-eddd-49b2-a167-c8219aa64d88" containerName="payload-pre-processing" Sep 07 10:55:14.804978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804925 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:55:14.804978 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.804937 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6efef0-a0bd-4fd3-abd9-9716086fdf02" containerName="revoke-keys" Sep 07 10:55:14.810097 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.810059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:14.812517 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.812494 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 10:55:14.813035 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.812785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 10:55:14.813860 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.813835 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6"] Sep 07 10:55:14.848125 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.848088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6\" (UID: \"0b7c4afa-9f6a-4841-988d-5680aef69055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:14.949207 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.949182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6\" (UID: \"0b7c4afa-9f6a-4841-988d-5680aef69055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:14.949818 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:14.949801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6\" (UID: \"0b7c4afa-9f6a-4841-988d-5680aef69055\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:15.123030 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:15.122954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:15.393228 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:15.393200 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6"] Sep 07 10:55:15.395118 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:55:15.395092 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7c4afa_9f6a_4841_988d_5680aef69055.slice/crio-0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67 WatchSource:0}: Error finding container 0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67: Status 404 returned error can't find the container with id 0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67 Sep 07 10:55:16.238177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:16.238143 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b7c4afa-9f6a-4841-988d-5680aef69055" containerID="a57a744441a68b1bf954cda427bb84f45c7fd391c451f33433abe52a9f2e75c1" exitCode=0 Sep 07 10:55:16.238558 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:16.238231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" event={"ID":"0b7c4afa-9f6a-4841-988d-5680aef69055","Type":"ContainerDied","Data":"a57a744441a68b1bf954cda427bb84f45c7fd391c451f33433abe52a9f2e75c1"} Sep 07 10:55:16.238558 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:16.238285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" event={"ID":"0b7c4afa-9f6a-4841-988d-5680aef69055","Type":"ContainerStarted","Data":"0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67"} Sep 07 10:55:17.385493 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:17.385471 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:17.475245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:17.475214 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca\") pod \"0b7c4afa-9f6a-4841-988d-5680aef69055\" (UID: \"0b7c4afa-9f6a-4841-988d-5680aef69055\") " Sep 07 10:55:17.475620 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:17.475577 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b7c4afa-9f6a-4841-988d-5680aef69055" (UID: "0b7c4afa-9f6a-4841-988d-5680aef69055"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:17.576736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:17.576658 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b7c4afa-9f6a-4841-988d-5680aef69055-maas-api-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:18.251649 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:18.251617 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" Sep 07 10:55:18.251814 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:18.251645 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6" event={"ID":"0b7c4afa-9f6a-4841-988d-5680aef69055","Type":"ContainerDied","Data":"0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67"} Sep 07 10:55:18.251814 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:18.251696 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b4fb0afdfa6a708aab2241120119c7f2e4539f62763cd684edd14f9146e0f67" Sep 07 10:55:25.306656 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:25.306623 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j"] Sep 07 10:55:25.309506 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:25.309481 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-5a0ff8-29812965-qzs2j"] Sep 07 10:55:25.498534 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:25.498497 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6"] Sep 07 10:55:25.500986 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:25.500961 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-5a0ff8-zgzz6"] Sep 07 10:55:26.723936 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:26.723898 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7c4afa-9f6a-4841-988d-5680aef69055" path="/var/lib/kubelet/pods/0b7c4afa-9f6a-4841-988d-5680aef69055/volumes" Sep 07 10:55:26.724316 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:26.724294 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd" path="/var/lib/kubelet/pods/7447a2d3-ab4c-4ee5-b3db-d5d026a47cbd/volumes" Sep 07 10:55:32.243230 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:32.243200 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:55:32.243659 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:32.243405 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" containerID="cri-o://176d41111cc97df5aee0221ed8ea07cf800bd9e7197b7247557b9bf594fad9a6" gracePeriod=30 Sep 07 10:55:33.131424 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:33.131392 2577 scope.go:117] "RemoveContainer" containerID="a4b34350e1c81dcf626d0900b6d7782c4f5ba3cecca5d620193283bf113269be" Sep 07 10:55:37.370536 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.370503 2577 generic.go:356] "Generic (PLEG): container finished" podID="1cb09564-22c0-4614-bec1-c4ed18543234" containerID="176d41111cc97df5aee0221ed8ea07cf800bd9e7197b7247557b9bf594fad9a6" exitCode=0 Sep 07 10:55:37.370922 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.370547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerDied","Data":"176d41111cc97df5aee0221ed8ea07cf800bd9e7197b7247557b9bf594fad9a6"} Sep 07 10:55:37.370922 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.370665 2577 scope.go:117] "RemoveContainer" containerID="55bd0aa8bf7237a2c5f34fe1a04924b95d99b0f80fce165049502260657a0969" Sep 07 10:55:37.494309 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.494285 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:55:37.557555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557490 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr9lw\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557531 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557555 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557551 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557576 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557658 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557706 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557772 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.557827 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557810 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.558088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557848 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:37.558088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.557877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket\") pod \"1cb09564-22c0-4614-bec1-c4ed18543234\" (UID: \"1cb09564-22c0-4614-bec1-c4ed18543234\") " Sep 07 10:55:37.558088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558006 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:37.558088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:37.558088 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558030 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:37.558510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558231 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.558510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558311 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.558510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558324 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cb09564-22c0-4614-bec1-c4ed18543234-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.558510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558335 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.558510 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.558391 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:37.559780 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.559758 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw" (OuterVolumeSpecName: "kube-api-access-wr9lw") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "kube-api-access-wr9lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:37.559914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.559901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:37.559991 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.559971 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:37.560028 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.559988 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cb09564-22c0-4614-bec1-c4ed18543234" (UID: "1cb09564-22c0-4614-bec1-c4ed18543234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:55:37.659636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.659614 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.659636 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.659632 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.659763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.659641 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9lw\" (UniqueName: \"kubernetes.io/projected/1cb09564-22c0-4614-bec1-c4ed18543234-kube-api-access-wr9lw\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.659763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.659649 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cb09564-22c0-4614-bec1-c4ed18543234-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:37.659763 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:37.659657 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cb09564-22c0-4614-bec1-c4ed18543234-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:38.380227 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.380201 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" Sep 07 10:55:38.380652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.380202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt" event={"ID":"1cb09564-22c0-4614-bec1-c4ed18543234","Type":"ContainerDied","Data":"c44c694ec122be3e83ef00a7fd95d4a109bed3617e66ca94d189c2967b974b90"} Sep 07 10:55:38.380652 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.380332 2577 scope.go:117] "RemoveContainer" containerID="176d41111cc97df5aee0221ed8ea07cf800bd9e7197b7247557b9bf594fad9a6" Sep 07 10:55:38.401699 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.401671 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:55:38.405261 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.405237 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5a0ff8-openshift-default-857db6fb4d-jqztt"] Sep 07 10:55:38.723678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:38.723652 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" path="/var/lib/kubelet/pods/1cb09564-22c0-4614-bec1-c4ed18543234/volumes" Sep 07 10:55:43.242519 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.242486 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:55:43.243091 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.242743 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" podUID="c7f44ca8-36be-4f74-ba35-7588bf735352" containerName="payload-processing" containerID="cri-o://25a39d8f7d670dab67f2de3835f52780dd4b8f81b005622d19a32431b3145727" gracePeriod=30 Sep 07 10:55:43.249320 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.249254 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:55:43.250799 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.249532 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" podUID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" containerName="payload-pre-processing" containerID="cri-o://bfbed057a8b05510edea221a23aa3b3e958545cf16f43c6bfd5dc5b5a242aced" gracePeriod=30 Sep 07 10:55:43.409461 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.409426 2577 generic.go:356] "Generic (PLEG): container finished" podID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" containerID="bfbed057a8b05510edea221a23aa3b3e958545cf16f43c6bfd5dc5b5a242aced" exitCode=0 Sep 07 10:55:43.409622 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.409504 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" event={"ID":"6b5f75ba-1f25-43d2-930c-1e6b614ee80d","Type":"ContainerDied","Data":"bfbed057a8b05510edea221a23aa3b3e958545cf16f43c6bfd5dc5b5a242aced"} Sep 07 10:55:43.411450 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.411424 2577 generic.go:356] "Generic (PLEG): container finished" podID="c7f44ca8-36be-4f74-ba35-7588bf735352" containerID="25a39d8f7d670dab67f2de3835f52780dd4b8f81b005622d19a32431b3145727" exitCode=0 Sep 07 10:55:43.411558 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.411464 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" event={"ID":"c7f44ca8-36be-4f74-ba35-7588bf735352","Type":"ContainerDied","Data":"25a39d8f7d670dab67f2de3835f52780dd4b8f81b005622d19a32431b3145727"} Sep 07 10:55:43.491537 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.491511 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:55:43.519440 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.519414 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:55:43.617012 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.616978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xwsx\" (UniqueName: \"kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx\") pod \"c7f44ca8-36be-4f74-ba35-7588bf735352\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " Sep 07 10:55:43.617186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.617040 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnm9b\" (UniqueName: \"kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b\") pod \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " Sep 07 10:55:43.617186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.617091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume\") pod \"c7f44ca8-36be-4f74-ba35-7588bf735352\" (UID: \"c7f44ca8-36be-4f74-ba35-7588bf735352\") " Sep 07 10:55:43.617186 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.617156 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume\") pod \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\" (UID: \"6b5f75ba-1f25-43d2-930c-1e6b614ee80d\") " Sep 07 10:55:43.617678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.617624 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b5f75ba-1f25-43d2-930c-1e6b614ee80d" (UID: "6b5f75ba-1f25-43d2-930c-1e6b614ee80d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:43.617678 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.617630 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7f44ca8-36be-4f74-ba35-7588bf735352" (UID: "c7f44ca8-36be-4f74-ba35-7588bf735352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:43.619626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.619564 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b" (OuterVolumeSpecName: "kube-api-access-dnm9b") pod "6b5f75ba-1f25-43d2-930c-1e6b614ee80d" (UID: "6b5f75ba-1f25-43d2-930c-1e6b614ee80d"). InnerVolumeSpecName "kube-api-access-dnm9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:43.619736 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.619668 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx" (OuterVolumeSpecName: "kube-api-access-9xwsx") pod "c7f44ca8-36be-4f74-ba35-7588bf735352" (UID: "c7f44ca8-36be-4f74-ba35-7588bf735352"). InnerVolumeSpecName "kube-api-access-9xwsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:43.718204 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.718179 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:43.718204 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.718201 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xwsx\" (UniqueName: \"kubernetes.io/projected/c7f44ca8-36be-4f74-ba35-7588bf735352-kube-api-access-9xwsx\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:43.718347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.718210 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnm9b\" (UniqueName: \"kubernetes.io/projected/6b5f75ba-1f25-43d2-930c-1e6b614ee80d-kube-api-access-dnm9b\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:43.718347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:43.718219 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f44ca8-36be-4f74-ba35-7588bf735352-plugins-config-volume\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:44.417778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.417741 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" Sep 07 10:55:44.417778 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.417748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9" event={"ID":"6b5f75ba-1f25-43d2-930c-1e6b614ee80d","Type":"ContainerDied","Data":"241d3a53ef1ffce3020e97b630cb41ce8646b92e0405ff06de7e80dd7a0a55ee"} Sep 07 10:55:44.418260 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.417816 2577 scope.go:117] "RemoveContainer" containerID="bfbed057a8b05510edea221a23aa3b3e958545cf16f43c6bfd5dc5b5a242aced" Sep 07 10:55:44.419214 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.419196 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" Sep 07 10:55:44.419442 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.419415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc" event={"ID":"c7f44ca8-36be-4f74-ba35-7588bf735352","Type":"ContainerDied","Data":"3df2419c50ae2b82dfc28ff25fa14b963e118cdba70b842c23bdec154da7ba7c"} Sep 07 10:55:44.427375 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.427353 2577 scope.go:117] "RemoveContainer" containerID="25a39d8f7d670dab67f2de3835f52780dd4b8f81b005622d19a32431b3145727" Sep 07 10:55:44.441514 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.441491 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:55:44.445310 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.445291 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-659410-7fcddb6b4b-qqkj9"] Sep 07 10:55:44.454914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.454891 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:55:44.458495 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.458476 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-659410-796c744bc-p9lwc"] Sep 07 10:55:44.722783 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.722753 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" path="/var/lib/kubelet/pods/6b5f75ba-1f25-43d2-930c-1e6b614ee80d/volumes" Sep 07 10:55:44.723114 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:44.723101 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f44ca8-36be-4f74-ba35-7588bf735352" path="/var/lib/kubelet/pods/c7f44ca8-36be-4f74-ba35-7588bf735352/volumes" Sep 07 10:55:49.918346 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:49.918312 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:55:49.918807 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:49.918525 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" containerID="cri-o://f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d" gracePeriod=30 Sep 07 10:55:52.559911 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.559876 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:55:52.560276 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.560068 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-597fd75c55-psqw6" podUID="79e0c667-7c49-4080-a032-80f47d969d87" containerName="manager" containerID="cri-o://7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda" gracePeriod=30 Sep 07 10:55:52.817704 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.817645 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:55:52.902899 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.902871 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbct6\" (UniqueName: \"kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6\") pod \"79e0c667-7c49-4080-a032-80f47d969d87\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " Sep 07 10:55:52.903087 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.902942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") pod \"79e0c667-7c49-4080-a032-80f47d969d87\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " Sep 07 10:55:52.903087 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.902998 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs\") pod \"79e0c667-7c49-4080-a032-80f47d969d87\" (UID: \"79e0c667-7c49-4080-a032-80f47d969d87\") " Sep 07 10:55:52.905039 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.905007 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6" (OuterVolumeSpecName: "kube-api-access-fbct6") pod "79e0c667-7c49-4080-a032-80f47d969d87" (UID: "79e0c667-7c49-4080-a032-80f47d969d87"). InnerVolumeSpecName "kube-api-access-fbct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:52.905039 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.905021 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79e0c667-7c49-4080-a032-80f47d969d87" (UID: "79e0c667-7c49-4080-a032-80f47d969d87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:55:52.905039 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:52.905007 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "79e0c667-7c49-4080-a032-80f47d969d87" (UID: "79e0c667-7c49-4080-a032-80f47d969d87"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:55:53.004448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.004422 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-metrics-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:53.004448 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.004444 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbct6\" (UniqueName: \"kubernetes.io/projected/79e0c667-7c49-4080-a032-80f47d969d87-kube-api-access-fbct6\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:53.004618 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.004453 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/79e0c667-7c49-4080-a032-80f47d969d87-webhook-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:53.469515 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.469482 2577 generic.go:356] "Generic (PLEG): container finished" podID="79e0c667-7c49-4080-a032-80f47d969d87" containerID="7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda" exitCode=0 Sep 07 10:55:53.469724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.469527 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-psqw6" Sep 07 10:55:53.469724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.469569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-psqw6" event={"ID":"79e0c667-7c49-4080-a032-80f47d969d87","Type":"ContainerDied","Data":"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda"} Sep 07 10:55:53.469724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.469644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-psqw6" event={"ID":"79e0c667-7c49-4080-a032-80f47d969d87","Type":"ContainerDied","Data":"2bf1742290d5e6524c4c32de26db6ecdff31617c5d0e0b3bf9f2f513a690813f"} Sep 07 10:55:53.469724 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.469674 2577 scope.go:117] "RemoveContainer" containerID="7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda" Sep 07 10:55:53.488090 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.488071 2577 scope.go:117] "RemoveContainer" containerID="7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda" Sep 07 10:55:53.488369 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:53.488349 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda\": container with ID starting with 7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda not found: ID does not exist" containerID="7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda" Sep 07 10:55:53.488428 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.488380 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda"} err="failed to get container status \"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda\": rpc error: code = NotFound desc = could not find container \"7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda\": container with ID starting with 7b7d7e712814437d5353015e51a2c05431b4bd825541fea607de3d639910ceda not found: ID does not exist" Sep 07 10:55:53.494848 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.494823 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:55:53.498749 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:53.498729 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-597fd75c55-psqw6"] Sep 07 10:55:54.723066 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:54.723033 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e0c667-7c49-4080-a032-80f47d969d87" path="/var/lib/kubelet/pods/79e0c667-7c49-4080-a032-80f47d969d87/volumes" Sep 07 10:55:55.177079 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.177056 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:55:55.222893 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.222866 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.222902 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.222931 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.222964 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb4zx\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223003 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223076 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223122 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:55.223343 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223215 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo\") pod \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\" (UID: \"c0d88a4e-9c01-4e74-8274-af54b8a47cbe\") " Sep 07 10:55:55.223540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223326 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:55.223540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223382 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 10:55:55.223540 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:55.223869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223566 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.223869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223623 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istiod-ca-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.223869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223638 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-workload-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.223869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223650 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-data\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.223869 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.223787 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:55.225324 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.225287 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 07 10:55:55.225499 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.225484 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 07 10:55:55.225602 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.225563 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:55.225668 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.225650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx" (OuterVolumeSpecName: "kube-api-access-kb4zx") pod "c0d88a4e-9c01-4e74-8274-af54b8a47cbe" (UID: "c0d88a4e-9c01-4e74-8274-af54b8a47cbe"). InnerVolumeSpecName "kube-api-access-kb4zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:55:55.324498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.324407 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-token\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.324498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.324437 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-podinfo\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.324498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.324447 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-istio-envoy\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.324498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.324455 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb4zx\" (UniqueName: \"kubernetes.io/projected/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-kube-api-access-kb4zx\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.324498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.324464 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d88a4e-9c01-4e74-8274-af54b8a47cbe-credential-socket\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:55:55.482463 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.482426 2577 generic.go:356] "Generic (PLEG): container finished" podID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerID="f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d" exitCode=0 Sep 07 10:55:55.482653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.482481 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" Sep 07 10:55:55.482653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.482484 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerDied","Data":"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d"} Sep 07 10:55:55.482653 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.482623 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f" event={"ID":"c0d88a4e-9c01-4e74-8274-af54b8a47cbe","Type":"ContainerDied","Data":"69049a81afe9e9024f1f01f7d80dc4bc029232be375c7902f45075ddab65a62f"} Sep 07 10:55:55.482798 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.482657 2577 scope.go:117] "RemoveContainer" containerID="f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d" Sep 07 10:55:55.491928 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.491907 2577 scope.go:117] "RemoveContainer" containerID="1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f" Sep 07 10:55:55.500298 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.500282 2577 scope.go:117] "RemoveContainer" containerID="f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d" Sep 07 10:55:55.500537 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:55.500516 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d\": container with ID starting with f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d not found: ID does not exist" containerID="f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d" Sep 07 10:55:55.500617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.500549 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d"} err="failed to get container status \"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d\": rpc error: code = NotFound desc = could not find container \"f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d\": container with ID starting with f8202e21eaab190321d0a926dbd272889663d931265f63ceb6a2b368b711795d not found: ID does not exist" Sep 07 10:55:55.500617 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.500571 2577 scope.go:117] "RemoveContainer" containerID="1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f" Sep 07 10:55:55.500848 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:55:55.500828 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f\": container with ID starting with 1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f not found: ID does not exist" containerID="1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f" Sep 07 10:55:55.500890 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.500856 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f"} err="failed to get container status \"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f\": rpc error: code = NotFound desc = could not find container \"1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f\": container with ID starting with 1055bcdd27d7ab1c3b120f5924b391af6179676b53cdc1d2a488da59f70cc12f not found: ID does not exist" Sep 07 10:55:55.503817 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.503798 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:55:55.507808 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:55.507789 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-659410-openshift-default-6578fb998d-qs58f"] Sep 07 10:55:56.723462 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:55:56.723428 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" path="/var/lib/kubelet/pods/c0d88a4e-9c01-4e74-8274-af54b8a47cbe/volumes" Sep 07 10:56:33.160460 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:33.160388 2577 scope.go:117] "RemoveContainer" containerID="7cedda4f93f932966c1d1c4673506674802c0e83f314acd428aac114aa5dc60a" Sep 07 10:56:54.342914 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.342880 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:56:54.343319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343304 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343319 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343319 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e0c667-7c49-4080-a032-80f47d969d87" containerName="manager" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343334 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e0c667-7c49-4080-a032-80f47d969d87" containerName="manager" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343345 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343352 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343365 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" containerName="payload-pre-processing" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343370 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" containerName="payload-pre-processing" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343377 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f44ca8-36be-4f74-ba35-7588bf735352" containerName="payload-processing" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343382 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f44ca8-36be-4f74-ba35-7588bf735352" containerName="payload-processing" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343389 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7c4afa-9f6a-4841-988d-5680aef69055" containerName="revoke-keys" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343395 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7c4afa-9f6a-4841-988d-5680aef69055" containerName="revoke-keys" Sep 07 10:56:54.343400 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343404 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343411 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343466 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343474 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343479 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5f75ba-1f25-43d2-930c-1e6b614ee80d" containerName="payload-pre-processing" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343484 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f44ca8-36be-4f74-ba35-7588bf735352" containerName="payload-processing" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343489 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb09564-22c0-4614-bec1-c4ed18543234" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343497 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="710528d9-1e2c-433d-ad68-b3c78a3bf660" containerName="istio-proxy" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343503 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7c4afa-9f6a-4841-988d-5680aef69055" containerName="revoke-keys" Sep 07 10:56:54.343765 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.343508 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e0c667-7c49-4080-a032-80f47d969d87" containerName="manager" Sep 07 10:56:54.346667 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.346647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.349086 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.349064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 07 10:56:54.349211 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.349084 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 07 10:56:54.349281 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.349155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-78xk4\"" Sep 07 10:56:54.349354 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.349205 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 07 10:56:54.357212 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.357189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:56:54.434163 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.434132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.434278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.434192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.434278 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.434267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxrs\" (UniqueName: \"kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.535629 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.535580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxrs\" (UniqueName: \"kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.535725 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.535661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.535769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.535725 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.538199 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.538173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.538288 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.538208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.543626 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.543575 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxrs\" (UniqueName: \"kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs\") pod \"maas-controller-597fd75c55-b7srk\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.659498 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.659442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:54.931458 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.931433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:56:54.933598 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:56:54.933552 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8e185d_4fa4_48ea_9973_ccf4485f1051.slice/crio-33fa4ca77d7f41158a7606f99b22a03479bd9c2ab8f8a24d1e184d938e994f59 WatchSource:0}: Error finding container 33fa4ca77d7f41158a7606f99b22a03479bd9c2ab8f8a24d1e184d938e994f59: Status 404 returned error can't find the container with id 33fa4ca77d7f41158a7606f99b22a03479bd9c2ab8f8a24d1e184d938e994f59 Sep 07 10:56:54.934960 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:54.934942 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 07 10:56:55.798269 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:55.798226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-b7srk" event={"ID":"6c8e185d-4fa4-48ea-9973-ccf4485f1051","Type":"ContainerStarted","Data":"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f"} Sep 07 10:56:55.798654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:55.798288 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-b7srk" event={"ID":"6c8e185d-4fa4-48ea-9973-ccf4485f1051","Type":"ContainerStarted","Data":"33fa4ca77d7f41158a7606f99b22a03479bd9c2ab8f8a24d1e184d938e994f59"} Sep 07 10:56:55.798654 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:55.798438 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:56:55.816152 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:56:55.816070 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-597fd75c55-b7srk" podStartSLOduration=1.18639731 podStartE2EDuration="1.816056939s" totalImagesPullingTime="629.659629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:56:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:56:55.814268717 +0000 UTC m=+2183.671958035" watchObservedRunningTime="2026-09-07 10:56:55.816056939 +0000 UTC m=+2183.673746254" Sep 07 10:57:06.805839 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:06.805809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:57:18.245189 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.245147 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:57:18.245768 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.245470 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-597fd75c55-b7srk" podUID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" containerName="manager" containerID="cri-o://fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f" gracePeriod=30 Sep 07 10:57:18.486347 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.486320 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:57:18.567200 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.567113 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert\") pod \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " Sep 07 10:57:18.567365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.567236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvxrs\" (UniqueName: \"kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs\") pod \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " Sep 07 10:57:18.567365 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.567294 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs\") pod \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\" (UID: \"6c8e185d-4fa4-48ea-9973-ccf4485f1051\") " Sep 07 10:57:18.569367 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.569320 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6c8e185d-4fa4-48ea-9973-ccf4485f1051" (UID: "6c8e185d-4fa4-48ea-9973-ccf4485f1051"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:57:18.569490 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.569401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs" (OuterVolumeSpecName: "kube-api-access-mvxrs") pod "6c8e185d-4fa4-48ea-9973-ccf4485f1051" (UID: "6c8e185d-4fa4-48ea-9973-ccf4485f1051"). InnerVolumeSpecName "kube-api-access-mvxrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 10:57:18.569657 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.569641 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c8e185d-4fa4-48ea-9973-ccf4485f1051" (UID: "6c8e185d-4fa4-48ea-9973-ccf4485f1051"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 10:57:18.668380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.668346 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxrs\" (UniqueName: \"kubernetes.io/projected/6c8e185d-4fa4-48ea-9973-ccf4485f1051-kube-api-access-mvxrs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:57:18.668380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.668375 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-metrics-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:57:18.668380 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.668385 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6c8e185d-4fa4-48ea-9973-ccf4485f1051-webhook-cert\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 10:57:18.928215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.928146 2577 generic.go:356] "Generic (PLEG): container finished" podID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" containerID="fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f" exitCode=0 Sep 07 10:57:18.928215 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.928188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-b7srk" Sep 07 10:57:18.928378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.928230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-b7srk" event={"ID":"6c8e185d-4fa4-48ea-9973-ccf4485f1051","Type":"ContainerDied","Data":"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f"} Sep 07 10:57:18.928378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.928284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-b7srk" event={"ID":"6c8e185d-4fa4-48ea-9973-ccf4485f1051","Type":"ContainerDied","Data":"33fa4ca77d7f41158a7606f99b22a03479bd9c2ab8f8a24d1e184d938e994f59"} Sep 07 10:57:18.928378 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.928305 2577 scope.go:117] "RemoveContainer" containerID="fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f" Sep 07 10:57:18.936917 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.936900 2577 scope.go:117] "RemoveContainer" containerID="fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f" Sep 07 10:57:18.937174 ip-10-0-141-95 kubenswrapper[2577]: E0907 10:57:18.937157 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f\": container with ID starting with fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f not found: ID does not exist" containerID="fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f" Sep 07 10:57:18.937223 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.937182 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f"} err="failed to get container status \"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f\": rpc error: code = NotFound desc = could not find container \"fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f\": container with ID starting with fa6b4e372cccef001a57010d65acd05cc9994f7efa47a36d148002d0fb1afd2f not found: ID does not exist" Sep 07 10:57:18.943213 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.943192 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:57:18.947241 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:18.947222 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-597fd75c55-b7srk"] Sep 07 10:57:19.719665 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.719633 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-597fd75c55-vsrg5"] Sep 07 10:57:19.720043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720009 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" containerName="manager" Sep 07 10:57:19.720043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720020 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" containerName="manager" Sep 07 10:57:19.720043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720038 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:57:19.720043 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720044 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:57:19.720177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720113 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d88a4e-9c01-4e74-8274-af54b8a47cbe" containerName="istio-proxy" Sep 07 10:57:19.720177 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.720121 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" containerName="manager" Sep 07 10:57:19.724469 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.724452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.726984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.726953 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 07 10:57:19.726984 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.726978 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-78xk4\"" Sep 07 10:57:19.727188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.726996 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 07 10:57:19.727188 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.727061 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 07 10:57:19.731663 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.731638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-vsrg5"] Sep 07 10:57:19.879245 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.879204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-webhook-cert\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.879453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.879324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-metrics-certs\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.879453 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.879356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqfw\" (UniqueName: \"kubernetes.io/projected/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-kube-api-access-xbqfw\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.980273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.980196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-metrics-certs\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.980273 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.980229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqfw\" (UniqueName: \"kubernetes.io/projected/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-kube-api-access-xbqfw\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.980484 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.980298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-webhook-cert\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.982919 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.982893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-webhook-cert\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.983038 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.982959 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-metrics-certs\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:19.988937 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:19.988915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqfw\" (UniqueName: \"kubernetes.io/projected/c5d60254-20f8-40b9-bcb4-e68ba72d6d06-kube-api-access-xbqfw\") pod \"maas-controller-597fd75c55-vsrg5\" (UID: \"c5d60254-20f8-40b9-bcb4-e68ba72d6d06\") " pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:20.035769 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.035737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:20.320476 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.320449 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-597fd75c55-vsrg5"] Sep 07 10:57:20.323153 ip-10-0-141-95 kubenswrapper[2577]: W0907 10:57:20.323126 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d60254_20f8_40b9_bcb4_e68ba72d6d06.slice/crio-fcca2c56a894c5db769b8c0fce3f504560e906a95658cb1c9677bbe707dd7633 WatchSource:0}: Error finding container fcca2c56a894c5db769b8c0fce3f504560e906a95658cb1c9677bbe707dd7633: Status 404 returned error can't find the container with id fcca2c56a894c5db769b8c0fce3f504560e906a95658cb1c9677bbe707dd7633 Sep 07 10:57:20.726321 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.726284 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c8e185d-4fa4-48ea-9973-ccf4485f1051" path="/var/lib/kubelet/pods/6c8e185d-4fa4-48ea-9973-ccf4485f1051/volumes" Sep 07 10:57:20.944528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.944032 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-vsrg5" event={"ID":"c5d60254-20f8-40b9-bcb4-e68ba72d6d06","Type":"ContainerStarted","Data":"be7bfa3d520e488dde9870e6b20fef82dbfd50e0d3ec58b260caf69717d93a70"} Sep 07 10:57:20.944528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.944123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:20.944528 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.944167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-597fd75c55-vsrg5" event={"ID":"c5d60254-20f8-40b9-bcb4-e68ba72d6d06","Type":"ContainerStarted","Data":"fcca2c56a894c5db769b8c0fce3f504560e906a95658cb1c9677bbe707dd7633"} Sep 07 10:57:20.960032 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:20.959938 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-597fd75c55-vsrg5" podStartSLOduration=1.4852740770000001 podStartE2EDuration="1.95991744s" totalImagesPullingTime="474.643363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:57:20.958886928 +0000 UTC m=+2208.816576246" watchObservedRunningTime="2026-09-07 10:57:20.95991744 +0000 UTC m=+2208.817606758" Sep 07 10:57:31.951903 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:31.951868 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-597fd75c55-vsrg5" Sep 07 10:57:33.197656 ip-10-0-141-95 kubenswrapper[2577]: I0907 10:57:33.197628 2577 scope.go:117] "RemoveContainer" containerID="1ce77dedc464913966ab563afa5a626f86f67102b7c96e2e5c05d60c93ed5d43" Sep 07 11:00:00.130375 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.130343 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp"] Sep 07 11:00:00.133606 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.133575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" Sep 07 11:00:00.135527 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.135503 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9clhg\"" Sep 07 11:00:00.140189 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.140166 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp"] Sep 07 11:00:00.145245 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.145222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" Sep 07 11:00:00.421620 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.421576 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp"] Sep 07 11:00:00.423820 ip-10-0-141-95 kubenswrapper[2577]: W0907 11:00:00.423790 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0e2e3b_9a3d_4572_85f4_911632ee62e3.slice/crio-8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037 WatchSource:0}: Error finding container 8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037: Status 404 returned error can't find the container with id 8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037 Sep 07 11:00:00.749260 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.749220 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" event={"ID":"4e0e2e3b-9a3d-4572-85f4-911632ee62e3","Type":"ContainerStarted","Data":"46289aad3067efd9679cea875cd487b9959f0eef80cca44c98e7e10c703976cb"} Sep 07 11:00:00.749479 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.749282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" event={"ID":"4e0e2e3b-9a3d-4572-85f4-911632ee62e3","Type":"ContainerStarted","Data":"8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037"} Sep 07 11:00:00.763805 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:00.763758 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" podStartSLOduration=0.763743344 podStartE2EDuration="763.743344ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:00:00.761973079 +0000 UTC m=+2368.619662395" watchObservedRunningTime="2026-09-07 11:00:00.763743344 +0000 UTC m=+2368.621432657" Sep 07 11:00:01.755122 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:01.755086 2577 generic.go:356] "Generic (PLEG): container finished" podID="4e0e2e3b-9a3d-4572-85f4-911632ee62e3" containerID="46289aad3067efd9679cea875cd487b9959f0eef80cca44c98e7e10c703976cb" exitCode=0 Sep 07 11:00:01.755493 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:01.755170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" event={"ID":"4e0e2e3b-9a3d-4572-85f4-911632ee62e3","Type":"ContainerDied","Data":"46289aad3067efd9679cea875cd487b9959f0eef80cca44c98e7e10c703976cb"} Sep 07 11:00:02.888361 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:02.888337 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" Sep 07 11:00:03.769510 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:03.769473 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" Sep 07 11:00:03.769510 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:03.769490 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812980-qxgzp" event={"ID":"4e0e2e3b-9a3d-4572-85f4-911632ee62e3","Type":"ContainerDied","Data":"8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037"} Sep 07 11:00:03.769796 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:03.769543 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e1125a417fe91b239709072f7601e8c64e9c4e881dc1d7331418f80a74ba037" Sep 07 11:00:33.236983 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:00:33.236948 2577 scope.go:117] "RemoveContainer" containerID="bbe2a3bcb29b5e9a7dd52a8af168ea4231cb10d15248489f7ab19cf1b6a53f70" Sep 07 11:01:33.256307 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.256271 2577 scope.go:117] "RemoveContainer" containerID="a57a744441a68b1bf954cda427bb84f45c7fd391c451f33433abe52a9f2e75c1" Sep 07 11:01:33.264894 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.264874 2577 scope.go:117] "RemoveContainer" containerID="142ec07e011e6f99f329392763ed62431f62670a8303290de4823beafa2765eb" Sep 07 11:01:33.893568 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.893528 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:33.894065 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.894047 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e0e2e3b-9a3d-4572-85f4-911632ee62e3" containerName="cleanup" Sep 07 11:01:33.894122 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.894067 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e0e2e3b-9a3d-4572-85f4-911632ee62e3" containerName="cleanup" Sep 07 11:01:33.894159 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.894140 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e0e2e3b-9a3d-4572-85f4-911632ee62e3" containerName="cleanup" Sep 07 11:01:33.899068 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.899047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:33.901811 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.901777 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 07 11:01:33.901932 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.901841 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 07 11:01:33.901932 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.901904 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bgjkj\"" Sep 07 11:01:33.902057 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.902013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 07 11:01:33.902057 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.902035 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 07 11:01:33.908278 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:33.908254 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:34.011821 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.011787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.011975 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.011844 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.011975 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.011929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.012051 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.012013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9dk\" (UniqueName: \"kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.113705 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.113656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.113874 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.113768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.113874 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.113811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.113874 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.113863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9dk\" (UniqueName: \"kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.114412 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.114382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.116464 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.116442 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.116616 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.116561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.121440 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.121407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9dk\" (UniqueName: \"kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk\") pod \"maas-api-58ff47f946-h9fzx\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.211760 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.211681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:34.487685 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:34.487661 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:34.490375 ip-10-0-141-95 kubenswrapper[2577]: W0907 11:01:34.490352 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951e25d7_75c5_475a_8556_5f29e1f12918.slice/crio-16094dda112ed5ff53bb938d1f78b09eab9d8080b4f8feef20905b04929cb5d7 WatchSource:0}: Error finding container 16094dda112ed5ff53bb938d1f78b09eab9d8080b4f8feef20905b04929cb5d7: Status 404 returned error can't find the container with id 16094dda112ed5ff53bb938d1f78b09eab9d8080b4f8feef20905b04929cb5d7 Sep 07 11:01:35.238942 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:35.238899 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" event={"ID":"951e25d7-75c5-475a-8556-5f29e1f12918","Type":"ContainerStarted","Data":"16094dda112ed5ff53bb938d1f78b09eab9d8080b4f8feef20905b04929cb5d7"} Sep 07 11:01:36.246105 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:36.246070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" event={"ID":"951e25d7-75c5-475a-8556-5f29e1f12918","Type":"ContainerStarted","Data":"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c"} Sep 07 11:01:36.246632 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:36.246136 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:36.262937 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:36.262850 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" podStartSLOduration=1.746272206 podStartE2EDuration="3.262836385s" totalImagesPullingTime="1.516564179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:36.261556681 +0000 UTC m=+2464.119245996" watchObservedRunningTime="2026-09-07 11:01:36.262836385 +0000 UTC m=+2464.120525700" Sep 07 11:01:41.888460 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.888421 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:41.888926 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.888693 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" podUID="951e25d7-75c5-475a-8556-5f29e1f12918" containerName="maas-api" containerID="cri-o://3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c" gracePeriod=30 Sep 07 11:01:41.893942 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.893921 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:41.920808 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.920784 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c"] Sep 07 11:01:41.924231 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.924212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:41.935221 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.935202 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c"] Sep 07 11:01:41.988835 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.988810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd086fb5-0254-415f-a6a1-f51c565a6389-openshift-service-ca\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:41.988974 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.988872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqwr\" (UniqueName: \"kubernetes.io/projected/cd086fb5-0254-415f-a6a1-f51c565a6389-kube-api-access-mkqwr\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:41.988974 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.988930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-maas-api-tls\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:41.989051 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:41.988991 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-metrics-certs\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.090623 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.090570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqwr\" (UniqueName: \"kubernetes.io/projected/cd086fb5-0254-415f-a6a1-f51c565a6389-kube-api-access-mkqwr\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.090779 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.090696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-maas-api-tls\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.090779 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.090761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-metrics-certs\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.090893 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.090831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd086fb5-0254-415f-a6a1-f51c565a6389-openshift-service-ca\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.091519 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.091492 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd086fb5-0254-415f-a6a1-f51c565a6389-openshift-service-ca\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.093343 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.093321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-maas-api-tls\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.093716 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.093701 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd086fb5-0254-415f-a6a1-f51c565a6389-metrics-certs\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.098833 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.098809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqwr\" (UniqueName: \"kubernetes.io/projected/cd086fb5-0254-415f-a6a1-f51c565a6389-kube-api-access-mkqwr\") pod \"maas-api-84bbc9dfb6-bm58c\" (UID: \"cd086fb5-0254-415f-a6a1-f51c565a6389\") " pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.145777 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.145732 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:42.191924 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.191902 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls\") pod \"951e25d7-75c5-475a-8556-5f29e1f12918\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " Sep 07 11:01:42.192009 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.191984 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca\") pod \"951e25d7-75c5-475a-8556-5f29e1f12918\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " Sep 07 11:01:42.192076 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.192062 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x9dk\" (UniqueName: \"kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk\") pod \"951e25d7-75c5-475a-8556-5f29e1f12918\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " Sep 07 11:01:42.192139 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.192126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs\") pod \"951e25d7-75c5-475a-8556-5f29e1f12918\" (UID: \"951e25d7-75c5-475a-8556-5f29e1f12918\") " Sep 07 11:01:42.192381 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.192349 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "951e25d7-75c5-475a-8556-5f29e1f12918" (UID: "951e25d7-75c5-475a-8556-5f29e1f12918"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 07 11:01:42.192507 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.192453 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/951e25d7-75c5-475a-8556-5f29e1f12918-openshift-service-ca\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 11:01:42.194320 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.194295 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "951e25d7-75c5-475a-8556-5f29e1f12918" (UID: "951e25d7-75c5-475a-8556-5f29e1f12918"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:01:42.194413 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.194318 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "951e25d7-75c5-475a-8556-5f29e1f12918" (UID: "951e25d7-75c5-475a-8556-5f29e1f12918"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 07 11:01:42.194413 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.194330 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk" (OuterVolumeSpecName: "kube-api-access-2x9dk") pod "951e25d7-75c5-475a-8556-5f29e1f12918" (UID: "951e25d7-75c5-475a-8556-5f29e1f12918"). InnerVolumeSpecName "kube-api-access-2x9dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 07 11:01:42.277845 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.277811 2577 generic.go:356] "Generic (PLEG): container finished" podID="951e25d7-75c5-475a-8556-5f29e1f12918" containerID="3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c" exitCode=0 Sep 07 11:01:42.277954 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.277862 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" Sep 07 11:01:42.277954 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.277903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" event={"ID":"951e25d7-75c5-475a-8556-5f29e1f12918","Type":"ContainerDied","Data":"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c"} Sep 07 11:01:42.277954 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.277948 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx" event={"ID":"951e25d7-75c5-475a-8556-5f29e1f12918","Type":"ContainerDied","Data":"16094dda112ed5ff53bb938d1f78b09eab9d8080b4f8feef20905b04929cb5d7"} Sep 07 11:01:42.278094 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.277970 2577 scope.go:117] "RemoveContainer" containerID="3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c" Sep 07 11:01:42.285904 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.285880 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:42.287359 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.287345 2577 scope.go:117] "RemoveContainer" containerID="3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c" Sep 07 11:01:42.287668 ip-10-0-141-95 kubenswrapper[2577]: E0907 11:01:42.287649 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c\": container with ID starting with 3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c not found: ID does not exist" containerID="3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c" Sep 07 11:01:42.287736 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.287676 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c"} err="failed to get container status \"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c\": rpc error: code = NotFound desc = could not find container \"3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c\": container with ID starting with 3bdba231f0b6e3d140cea3c29181c4087cb684d18c00dc6ff047386cf0f4cf3c not found: ID does not exist" Sep 07 11:01:42.293088 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.293068 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9dk\" (UniqueName: \"kubernetes.io/projected/951e25d7-75c5-475a-8556-5f29e1f12918-kube-api-access-2x9dk\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 11:01:42.293163 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.293092 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-metrics-certs\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 11:01:42.293163 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.293107 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/951e25d7-75c5-475a-8556-5f29e1f12918-maas-api-tls\") on node \"ip-10-0-141-95.ec2.internal\" DevicePath \"\"" Sep 07 11:01:42.299369 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.299350 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:42.303407 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.303375 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-58ff47f946-h9fzx"] Sep 07 11:01:42.559555 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.559528 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c"] Sep 07 11:01:42.562036 ip-10-0-141-95 kubenswrapper[2577]: W0907 11:01:42.562006 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd086fb5_0254_415f_a6a1_f51c565a6389.slice/crio-0d71d84cadfe45980c5d76d3fc435578598432cdad1ea890cfb8d4a612615508 WatchSource:0}: Error finding container 0d71d84cadfe45980c5d76d3fc435578598432cdad1ea890cfb8d4a612615508: Status 404 returned error can't find the container with id 0d71d84cadfe45980c5d76d3fc435578598432cdad1ea890cfb8d4a612615508 Sep 07 11:01:42.723439 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:42.723412 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951e25d7-75c5-475a-8556-5f29e1f12918" path="/var/lib/kubelet/pods/951e25d7-75c5-475a-8556-5f29e1f12918/volumes" Sep 07 11:01:43.286011 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:43.285973 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" event={"ID":"cd086fb5-0254-415f-a6a1-f51c565a6389","Type":"ContainerStarted","Data":"fdc4f9bc665c0151f8280e2eb072fd186082a6859897f5ea9bbf438dab8c490d"} Sep 07 11:01:43.286457 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:43.286038 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" event={"ID":"cd086fb5-0254-415f-a6a1-f51c565a6389","Type":"ContainerStarted","Data":"0d71d84cadfe45980c5d76d3fc435578598432cdad1ea890cfb8d4a612615508"} Sep 07 11:01:43.286457 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:43.286078 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:01:43.303407 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:43.303354 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" podStartSLOduration=1.889366771 podStartE2EDuration="2.30333668s" totalImagesPullingTime="413.969909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 11:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:01:43.300381972 +0000 UTC m=+2471.158071289" watchObservedRunningTime="2026-09-07 11:01:43.30333668 +0000 UTC m=+2471.161025992" Sep 07 11:01:49.297393 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:01:49.297359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84bbc9dfb6-bm58c" Sep 07 11:02:14.286146 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:14.286097 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-597fd75c55-vsrg5_c5d60254-20f8-40b9-bcb4-e68ba72d6d06/manager/0.log" Sep 07 11:02:14.440392 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:14.440331 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fvdcj_8d987cce-39d5-4cac-ba92-4fba3d49fb71/server/0.log" Sep 07 11:02:14.582718 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:14.582650 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xph9l_577b010a-6084-440b-b873-62af7d015b08/manager/0.log" Sep 07 11:02:15.055060 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:15.055037 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-jc5vc_87d3670a-51d7-4798-ae27-eeddba6b32d2/copy-manifests/0.log" Sep 07 11:02:15.062100 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:15.062081 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-jc5vc_87d3670a-51d7-4798-ae27-eeddba6b32d2/manager/0.log" Sep 07 11:02:15.454725 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:15.454649 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84bbc9dfb6-bm58c_cd086fb5-0254-415f-a6a1-f51c565a6389/maas-api/0.log" Sep 07 11:02:15.718091 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:15.718015 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29812965-5lc6n_08d40f88-c523-4a1a-a20e-0c6127455b0d/cleanup/0.log" Sep 07 11:02:15.840986 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:15.840961 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29812980-qxgzp_4e0e2e3b-9a3d-4572-85f4-911632ee62e3/cleanup/0.log" Sep 07 11:02:18.569502 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:18.569475 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49_ffe2c7b7-4459-4f6a-aaac-a672306eefd3/istio-proxy/0.log" Sep 07 11:02:18.719726 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:18.719698 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-56ffj_4c98e54c-c885-47ad-b34d-e2ba075a7908/discovery/0.log" Sep 07 11:02:19.120353 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:19.120324 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/1.log" Sep 07 11:02:19.266115 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:19.266090 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-cp4dv_2297757b-3f4d-4aed-b1c6-1537057d0888/payload-pre-processing/0.log" Sep 07 11:02:19.440185 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:19.440096 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 11:02:20.015211 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:20.015172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5vvnv_79008255-a57f-421d-85b4-7089ed5ba0ab/main/0.log" Sep 07 11:02:30.299294 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:30.299195 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b5fbr_eb112766-fde8-43ff-9a61-36845bf0f30f/global-pull-secret-syncer/0.log" Sep 07 11:02:30.506068 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:30.506036 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hvgr4_2fd4bdbf-9b94-4d89-899b-1b4fd6018932/konnectivity-agent/0.log" Sep 07 11:02:30.552797 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:30.552728 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-95.ec2.internal_17f670fc079e23e62690da1f2f92b580/haproxy/0.log" Sep 07 11:02:37.036247 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.036218 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/alertmanager/0.log" Sep 07 11:02:37.062932 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.062907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/config-reloader/0.log" Sep 07 11:02:37.089676 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.089633 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/kube-rbac-proxy-web/0.log" Sep 07 11:02:37.111226 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.111202 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/kube-rbac-proxy/0.log" Sep 07 11:02:37.133138 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.133110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/kube-rbac-proxy-metric/0.log" Sep 07 11:02:37.154385 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.154359 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/prom-label-proxy/0.log" Sep 07 11:02:37.177115 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.177091 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5366bba2-fa89-49f6-b869-08ef034d58ff/init-config-reloader/0.log" Sep 07 11:02:37.246256 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.246216 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7ppg8_e6d78436-0df6-4525-a79b-dfe5a2814eb5/kube-state-metrics/0.log" Sep 07 11:02:37.265728 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.265699 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7ppg8_e6d78436-0df6-4525-a79b-dfe5a2814eb5/kube-rbac-proxy-main/0.log" Sep 07 11:02:37.294325 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.294257 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-7ppg8_e6d78436-0df6-4525-a79b-dfe5a2814eb5/kube-rbac-proxy-self/0.log" Sep 07 11:02:37.323301 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.323277 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5559b5685-dbj7z_e9944e5c-b7cc-4865-abee-cc5393a49024/metrics-server/0.log" Sep 07 11:02:37.349969 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.349944 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-8j2z5_0bec0a65-32ef-41e0-8931-7c46306d0139/monitoring-plugin/0.log" Sep 07 11:02:37.381853 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.381823 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k6fx_ca7a9454-5371-4477-8efd-2b6ddf3c2fa2/node-exporter/0.log" Sep 07 11:02:37.408303 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.408279 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k6fx_ca7a9454-5371-4477-8efd-2b6ddf3c2fa2/kube-rbac-proxy/0.log" Sep 07 11:02:37.433353 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.433330 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4k6fx_ca7a9454-5371-4477-8efd-2b6ddf3c2fa2/init-textfile/0.log" Sep 07 11:02:37.654640 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.654551 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fs698_33e8e89f-9489-46e7-b15f-1c9b5f33be15/kube-rbac-proxy-main/0.log" Sep 07 11:02:37.691333 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.691312 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fs698_33e8e89f-9489-46e7-b15f-1c9b5f33be15/kube-rbac-proxy-self/0.log" Sep 07 11:02:37.718221 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.718208 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-fs698_33e8e89f-9489-46e7-b15f-1c9b5f33be15/openshift-state-metrics/0.log" Sep 07 11:02:37.758810 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.758782 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/prometheus/0.log" Sep 07 11:02:37.783568 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.783549 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/config-reloader/0.log" Sep 07 11:02:37.810465 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.810441 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/thanos-sidecar/0.log" Sep 07 11:02:37.839193 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.839172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/kube-rbac-proxy-web/0.log" Sep 07 11:02:37.864170 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.864145 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/kube-rbac-proxy/0.log" Sep 07 11:02:37.889288 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.889265 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/kube-rbac-proxy-thanos/0.log" Sep 07 11:02:37.912887 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.912819 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_bd89dd7b-df6a-4d46-a130-45c53d304fdd/init-config-reloader/0.log" Sep 07 11:02:37.950984 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.950963 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-d86sm_1e94ffbd-080c-4f67-8746-f3e8b192d286/prometheus-operator/0.log" Sep 07 11:02:37.972274 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:37.972252 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-d86sm_1e94ffbd-080c-4f67-8746-f3e8b192d286/kube-rbac-proxy/0.log" Sep 07 11:02:38.068166 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.068142 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78c745d5f8-45r9t_b9bc4b8e-17a5-4649-b4f0-729d3ec3c199/telemeter-client/0.log" Sep 07 11:02:38.142262 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.142233 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78c745d5f8-45r9t_b9bc4b8e-17a5-4649-b4f0-729d3ec3c199/reload/0.log" Sep 07 11:02:38.198671 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.198611 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78c745d5f8-45r9t_b9bc4b8e-17a5-4649-b4f0-729d3ec3c199/kube-rbac-proxy/0.log" Sep 07 11:02:38.236421 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.236398 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/thanos-query/0.log" Sep 07 11:02:38.259391 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.259371 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/kube-rbac-proxy-web/0.log" Sep 07 11:02:38.282080 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.282064 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/kube-rbac-proxy/0.log" Sep 07 11:02:38.306406 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.306387 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/prom-label-proxy/0.log" Sep 07 11:02:38.328323 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.328307 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/kube-rbac-proxy-rules/0.log" Sep 07 11:02:38.352394 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:38.352376 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-56775896b9-d2rjp_fe8656cf-c8d0-4010-b6f7-8bd8fccc53b7/kube-rbac-proxy-metrics/0.log" Sep 07 11:02:40.046950 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.046913 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h"] Sep 07 11:02:40.047526 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.047446 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951e25d7-75c5-475a-8556-5f29e1f12918" containerName="maas-api" Sep 07 11:02:40.047526 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.047466 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="951e25d7-75c5-475a-8556-5f29e1f12918" containerName="maas-api" Sep 07 11:02:40.047710 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.047607 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="951e25d7-75c5-475a-8556-5f29e1f12918" containerName="maas-api" Sep 07 11:02:40.050466 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.050449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.052361 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.052336 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxx4\"/\"kube-root-ca.crt\"" Sep 07 11:02:40.052475 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.052402 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jhxx4\"/\"default-dockercfg-jnqbs\"" Sep 07 11:02:40.052812 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.052795 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jhxx4\"/\"openshift-service-ca.crt\"" Sep 07 11:02:40.060566 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.060544 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h"] Sep 07 11:02:40.148436 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.148387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-podres\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.148436 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.148433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwj94\" (UniqueName: \"kubernetes.io/projected/85866793-1284-4c8d-b21f-ef6ca9e362a9-kube-api-access-vwj94\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.148712 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.148471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-sys\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.148712 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.148494 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-lib-modules\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.148712 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.148512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-proc\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.249869 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.249825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-podres\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.249869 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.249872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwj94\" (UniqueName: \"kubernetes.io/projected/85866793-1284-4c8d-b21f-ef6ca9e362a9-kube-api-access-vwj94\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.249910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-sys\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.249934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-lib-modules\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.249952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-proc\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.250013 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-podres\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.250025 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-sys\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.250090 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-proc\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.250136 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.250116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85866793-1284-4c8d-b21f-ef6ca9e362a9-lib-modules\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.257683 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.257653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwj94\" (UniqueName: \"kubernetes.io/projected/85866793-1284-4c8d-b21f-ef6ca9e362a9-kube-api-access-vwj94\") pod \"perf-node-gather-daemonset-66b4h\" (UID: \"85866793-1284-4c8d-b21f-ef6ca9e362a9\") " pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.361918 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.361786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:40.636309 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:40.636273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h"] Sep 07 11:02:40.638466 ip-10-0-141-95 kubenswrapper[2577]: W0907 11:02:40.638432 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85866793_1284_4c8d_b21f_ef6ca9e362a9.slice/crio-147c5ff4d3d284516483b860e523bbfed72d2bdc3c6b79fcdbf3730313bd319a WatchSource:0}: Error finding container 147c5ff4d3d284516483b860e523bbfed72d2bdc3c6b79fcdbf3730313bd319a: Status 404 returned error can't find the container with id 147c5ff4d3d284516483b860e523bbfed72d2bdc3c6b79fcdbf3730313bd319a Sep 07 11:02:41.579151 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:41.579112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" event={"ID":"85866793-1284-4c8d-b21f-ef6ca9e362a9","Type":"ContainerStarted","Data":"f54576dfaa8f1b00a341b7a0268e4d9b5b21ebaed3d46d4628e04187f75b5384"} Sep 07 11:02:41.579560 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:41.579172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" event={"ID":"85866793-1284-4c8d-b21f-ef6ca9e362a9","Type":"ContainerStarted","Data":"147c5ff4d3d284516483b860e523bbfed72d2bdc3c6b79fcdbf3730313bd319a"} Sep 07 11:02:41.579560 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:41.579234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:41.593637 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:41.593559 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" podStartSLOduration=1.593545314 podStartE2EDuration="1.593545314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 11:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 11:02:41.59278291 +0000 UTC m=+2529.450472227" watchObservedRunningTime="2026-09-07 11:02:41.593545314 +0000 UTC m=+2529.451234631" Sep 07 11:02:42.816739 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:42.816706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fn7dn_1c139569-281b-487b-9851-616f975116b1/dns/0.log" Sep 07 11:02:42.847327 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:42.847300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fn7dn_1c139569-281b-487b-9851-616f975116b1/kube-rbac-proxy/0.log" Sep 07 11:02:42.900717 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:42.900692 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lkh99_ae088060-c534-4d00-a2c8-40beda7a51b8/dns-node-resolver/0.log" Sep 07 11:02:43.518159 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:43.518130 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xz99t_dfde7950-68a7-43e5-b58e-fb40830b9c22/node-ca/0.log" Sep 07 11:02:44.469985 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.469950 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49_ffe2c7b7-4459-4f6a-aaac-a672306eefd3/istio-proxy/0.log" Sep 07 11:02:44.747502 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.747423 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-56ffj_4c98e54c-c885-47ad-b34d-e2ba075a7908/discovery/0.log" Sep 07 11:02:44.831392 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.831364 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/0.log" Sep 07 11:02:44.841557 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.841536 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/1.log" Sep 07 11:02:44.873209 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.873179 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-cp4dv_2297757b-3f4d-4aed-b1c6-1537057d0888/payload-pre-processing/0.log" Sep 07 11:02:44.907312 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:44.907286 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 11:02:45.780642 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:45.780604 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cdcwk_af0133f0-b811-47d0-b008-bab7a80516b9/serve-healthcheck-canary/0.log" Sep 07 11:02:46.475473 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:46.475450 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbrkj_0b9a1cae-7394-4c14-a23b-4c1d35f5d258/kube-rbac-proxy/0.log" Sep 07 11:02:46.495638 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:46.495612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbrkj_0b9a1cae-7394-4c14-a23b-4c1d35f5d258/exporter/0.log" Sep 07 11:02:46.515848 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:46.515827 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xbrkj_0b9a1cae-7394-4c14-a23b-4c1d35f5d258/extractor/0.log" Sep 07 11:02:47.596753 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:47.596721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jhxx4/perf-node-gather-daemonset-66b4h" Sep 07 11:02:49.026534 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:49.026488 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-597fd75c55-vsrg5_c5d60254-20f8-40b9-bcb4-e68ba72d6d06/manager/0.log" Sep 07 11:02:49.044709 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:49.044673 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fvdcj_8d987cce-39d5-4cac-ba92-4fba3d49fb71/server/0.log" Sep 07 11:02:49.122948 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:49.122916 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-xph9l_577b010a-6084-440b-b873-62af7d015b08/manager/0.log" Sep 07 11:02:49.290178 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:49.290085 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-jc5vc_87d3670a-51d7-4798-ae27-eeddba6b32d2/manager/0.log" Sep 07 11:02:49.310497 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:49.310469 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f5bfcc886-jc5vc_87d3670a-51d7-4798-ae27-eeddba6b32d2/copy-manifests/0.log" Sep 07 11:02:50.701008 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:50.700968 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-547fb475d5-8lrwn_b3983afd-47b6-4621-994a-f69c7784cfd4/manager/0.log" Sep 07 11:02:50.729681 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:50.729656 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-x4w68_10431c8d-ba99-4946-b33a-1f238c7d8362/openshift-lws-operator/0.log" Sep 07 11:02:50.768219 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:50.768195 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49_ffe2c7b7-4459-4f6a-aaac-a672306eefd3/istio-proxy/0.log" Sep 07 11:02:51.067028 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:51.066989 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-56ffj_4c98e54c-c885-47ad-b34d-e2ba075a7908/discovery/0.log" Sep 07 11:02:51.130943 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:51.130907 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/0.log" Sep 07 11:02:51.141339 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:51.141295 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/1.log" Sep 07 11:02:51.165974 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:51.165865 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-cp4dv_2297757b-3f4d-4aed-b1c6-1537057d0888/payload-pre-processing/0.log" Sep 07 11:02:51.195969 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:51.195934 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 11:02:52.699269 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:52.699238 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-mjn49_ffe2c7b7-4459-4f6a-aaac-a672306eefd3/istio-proxy/0.log" Sep 07 11:02:53.005648 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:53.005612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-56ffj_4c98e54c-c885-47ad-b34d-e2ba075a7908/discovery/0.log" Sep 07 11:02:53.061873 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:53.061839 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/0.log" Sep 07 11:02:53.071689 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:53.071611 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-s2jpw_16b4cfa1-20f7-42a8-83a3-213f7be5cbb6/istio-proxy/1.log" Sep 07 11:02:53.094188 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:53.094159 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-cp4dv_2297757b-3f4d-4aed-b1c6-1537057d0888/payload-pre-processing/0.log" Sep 07 11:02:53.119909 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:53.119877 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xhh2r_f4328b5b-7c17-4f21-b3a5-5e890b59fbb1/payload-processing/0.log" Sep 07 11:02:57.254359 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.254324 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/kube-multus-additional-cni-plugins/0.log" Sep 07 11:02:57.278621 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.278545 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/egress-router-binary-copy/0.log" Sep 07 11:02:57.306449 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.306420 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/cni-plugins/0.log" Sep 07 11:02:57.333944 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.333918 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/bond-cni-plugin/0.log" Sep 07 11:02:57.357289 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.357238 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/routeoverride-cni/0.log" Sep 07 11:02:57.381728 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.381692 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/whereabouts-cni-bincopy/0.log" Sep 07 11:02:57.402919 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.402891 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-94q5n_c3f94753-ba3e-411b-838c-0760af56c256/whereabouts-cni/0.log" Sep 07 11:02:57.793726 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.793688 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-pbklr_a9228703-6990-4c3f-b627-7597c3a0ff9e/kube-multus/0.log" Sep 07 11:02:57.943049 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.943014 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z5f5r_4b2886c9-c467-41ea-aa26-db0748ea4400/network-metrics-daemon/0.log" Sep 07 11:02:57.960881 ip-10-0-141-95 kubenswrapper[2577]: I0907 11:02:57.960848 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-z5f5r_4b2886c9-c467-41ea-aa26-db0748ea4400/kube-rbac-proxy/0.log"