Sep 16 04:46:58.467301 ip-10-0-142-144 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 04:46:58.467312 ip-10-0-142-144 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 04:46:58.467322 ip-10-0-142-144 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 04:46:58.467570 ip-10-0-142-144 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 04:47:08.716340 ip-10-0-142-144 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 04:47:08.716358 ip-10-0-142-144 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a3d46d6114e44d12a098ec80c2969a58 -- Sep 16 04:48:54.372872 ip-10-0-142-144 systemd[1]: Starting Kubernetes Kubelet... Sep 16 04:48:54.806856 ip-10-0-142-144 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 04:48:54.806856 ip-10-0-142-144 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 04:48:54.806856 ip-10-0-142-144 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 04:48:54.806856 ip-10-0-142-144 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 04:48:54.806856 ip-10-0-142-144 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 04:48:54.807532 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.806910 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 04:48:54.812808 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812792 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.812808 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812807 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812811 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812814 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812818 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812821 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812824 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812826 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812829 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812832 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812836 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812839 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812841 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812844 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812847 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812850 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812852 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812855 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812858 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812864 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812867 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.812875 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812870 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812873 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812876 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812878 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812883 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812887 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812890 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812893 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812896 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812899 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812901 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812904 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812906 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812909 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812911 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812914 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812917 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812919 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812921 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.813351 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812924 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812927 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812931 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812935 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812938 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812940 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812943 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812945 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812948 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812950 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812953 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812956 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812960 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812963 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812966 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812970 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812973 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812976 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812979 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.813866 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812982 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812985 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812988 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812991 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812993 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812996 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.812999 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813001 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813004 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813007 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813010 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813012 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813015 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813018 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813021 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813023 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813026 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813028 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813031 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813034 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.814339 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813036 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813039 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813041 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813044 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813046 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813049 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813052 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813055 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813059 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813061 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813064 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813067 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813069 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813072 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813074 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813077 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813079 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813083 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813086 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813088 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.815033 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813779 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813786 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813789 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813791 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813795 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813798 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813800 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813803 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813806 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813809 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813811 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813814 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813816 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813819 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813821 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813824 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813827 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813830 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813834 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813838 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.815695 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813841 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813843 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813846 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813848 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813851 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813853 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813856 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813859 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813861 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813864 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813866 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813869 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813872 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813874 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813877 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813880 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813883 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813885 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813889 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813891 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.816243 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813894 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813897 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813900 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813902 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813905 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813907 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813911 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813915 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813918 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813921 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813924 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813927 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813930 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813933 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813936 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813939 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813941 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813943 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813946 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.816786 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813948 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813951 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813954 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813957 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813959 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813962 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813964 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813967 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813969 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813972 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813975 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813977 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813980 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813982 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813985 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813988 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813990 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813993 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813995 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.817251 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.813998 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814002 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814006 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814009 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814012 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814015 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814018 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814021 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814024 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814027 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814030 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814032 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814035 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814037 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814040 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814043 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814046 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814048 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814051 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.817736 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814053 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814056 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814765 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814776 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814781 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814786 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814791 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814795 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814799 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814803 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814807 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814818 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814823 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814827 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814831 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814835 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814839 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814843 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814848 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.818223 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814855 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814870 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814874 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814881 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814887 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814896 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814901 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814906 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814910 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814914 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814919 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814923 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814928 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814933 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814937 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814941 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814946 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814955 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814960 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814964 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.818712 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814969 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814973 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814978 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814984 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814988 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814993 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.814998 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815002 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815007 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815016 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815020 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815024 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815030 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815034 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815040 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815044 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815049 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815053 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815057 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815061 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.819203 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815066 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815070 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815079 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815083 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815088 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815094 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815100 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815105 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815110 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815114 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815119 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815124 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815128 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815132 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815141 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815146 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815151 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815155 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815160 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.819725 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815164 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815170 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815175 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815179 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815183 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815188 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815199 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815204 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815209 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815214 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815218 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815222 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815226 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815231 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815235 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815239 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815243 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815247 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815251 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815259 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.820193 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815263 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815267 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.815272 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815586 2560 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815643 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815662 2560 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815668 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815675 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815680 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815686 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815692 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815697 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815702 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815707 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815712 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815717 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815722 2560 flags.go:64] FLAG: --cgroup-root="" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815727 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815731 2560 flags.go:64] FLAG: --client-ca-file="" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815736 2560 flags.go:64] FLAG: --cloud-provider="external" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815740 2560 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815748 2560 flags.go:64] FLAG: --cluster-domain="" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815752 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 04:48:54.820717 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815755 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815758 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815762 2560 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815766 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815769 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815772 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815775 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815778 2560 flags.go:64] FLAG: --contention-profiling="false" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815781 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815784 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815787 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815790 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815795 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815799 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815804 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815808 2560 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815812 2560 flags.go:64] FLAG: --enable-server="true" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815816 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815820 2560 flags.go:64] FLAG: --event-burst="100" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815824 2560 flags.go:64] FLAG: --event-qps="50" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815827 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815830 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815833 2560 flags.go:64] FLAG: --eviction-hard="" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815837 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 04:48:54.821307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815839 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815842 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815845 2560 flags.go:64] FLAG: --eviction-soft="" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815848 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815851 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815854 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815857 2560 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815860 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815863 2560 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815866 2560 flags.go:64] FLAG: --feature-gates="" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815869 2560 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815872 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815876 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815879 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815882 2560 flags.go:64] FLAG: --healthz-port="10248" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815884 2560 flags.go:64] FLAG: --help="false" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815887 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-142-144.ec2.internal" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815891 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815894 2560 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815896 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815901 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815905 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815908 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 04:48:54.821899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815911 2560 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815914 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815917 2560 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815920 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815923 2560 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815926 2560 flags.go:64] FLAG: --kube-reserved="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815930 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815932 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815935 2560 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815938 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815941 2560 flags.go:64] FLAG: --lock-file="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815944 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815947 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815950 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815956 2560 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815959 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815963 2560 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815965 2560 flags.go:64] FLAG: --logging-format="text" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815968 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815972 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815975 2560 flags.go:64] FLAG: --manifest-url="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815977 2560 flags.go:64] FLAG: --manifest-url-header="" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815982 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815985 2560 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815988 2560 flags.go:64] FLAG: --max-pods="110" Sep 16 04:48:54.822486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815991 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815994 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.815997 2560 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816000 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816003 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816007 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816010 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816018 2560 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816021 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816025 2560 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816028 2560 flags.go:64] FLAG: --pod-cidr="" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816031 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816036 2560 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816039 2560 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816042 2560 flags.go:64] FLAG: --pods-per-core="0" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816045 2560 flags.go:64] FLAG: --port="10250" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816048 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816050 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00136599b3f2c5acb" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816054 2560 flags.go:64] FLAG: --qos-reserved="" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816057 2560 flags.go:64] FLAG: --read-only-port="10255" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816060 2560 flags.go:64] FLAG: --register-node="true" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816063 2560 flags.go:64] FLAG: --register-with-taints="" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816066 2560 flags.go:64] FLAG: --registry-burst="10" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816070 2560 flags.go:64] FLAG: --registry-qps="5" Sep 16 04:48:54.823171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816073 2560 flags.go:64] FLAG: --reserved-cpus="" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816076 2560 flags.go:64] FLAG: --reserved-memory="" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816079 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816082 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816085 2560 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816088 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816091 2560 flags.go:64] FLAG: --runonce="false" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816093 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816097 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816100 2560 flags.go:64] FLAG: --seccomp-default="false" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816103 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816106 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816111 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816116 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816119 2560 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816122 2560 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816124 2560 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816127 2560 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816130 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816133 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816136 2560 flags.go:64] FLAG: --system-cgroups="" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816139 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816145 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816147 2560 flags.go:64] FLAG: --tls-cert-file="" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816150 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 04:48:54.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816155 2560 flags.go:64] FLAG: --tls-min-version="" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816157 2560 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816160 2560 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816163 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816166 2560 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816169 2560 flags.go:64] FLAG: --v="2" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816173 2560 flags.go:64] FLAG: --version="false" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816176 2560 flags.go:64] FLAG: --vmodule="" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816180 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816184 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816337 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816341 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816344 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816349 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816353 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816356 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816360 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816363 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816365 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816370 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816375 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816378 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.824429 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816381 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816392 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816395 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816398 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816400 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816403 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816406 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816409 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816411 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816414 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816417 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816419 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816422 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816425 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816427 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816430 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816432 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816435 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816437 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816441 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.824986 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816444 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816447 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816449 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816452 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816455 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816457 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816460 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816462 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816465 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816469 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816472 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816475 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816477 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816480 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816483 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816485 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816488 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816491 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816494 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816496 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.825477 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816499 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816501 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816504 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816507 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816510 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816513 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816515 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816518 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816521 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816523 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816526 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816528 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816531 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816534 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816536 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816539 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816543 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816561 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816565 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.825981 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816568 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816571 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816575 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816581 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816583 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816586 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816589 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816592 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816595 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816597 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816600 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816602 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816605 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816608 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816610 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816614 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816616 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816619 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816622 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816624 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.826440 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816627 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816630 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816633 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816635 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816638 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816642 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816646 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.816650 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.816655 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.824226 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.824351 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.824368 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.824374 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824437 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.826948 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824442 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824446 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824449 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824452 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824456 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824460 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824463 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824466 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824469 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824472 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824475 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824478 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824481 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824483 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824486 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824489 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824492 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824494 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824497 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.827313 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824500 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824502 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824505 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824508 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824510 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824514 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824516 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824519 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824521 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824525 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824529 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824533 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824536 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824539 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824542 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824558 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824562 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824565 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824567 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.827805 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824571 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824574 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824576 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824579 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824582 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824585 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824587 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824590 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824593 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824595 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824598 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824601 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824604 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824606 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824609 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824611 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824614 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824617 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824619 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824622 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.828289 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824625 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824628 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824630 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824634 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824637 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824639 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824642 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824645 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824647 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824650 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824652 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824655 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824658 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824661 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824663 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824666 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824668 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824671 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824674 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824677 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.828826 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824679 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824682 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824684 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824687 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824689 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824692 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824695 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824697 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824700 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824702 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824705 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824708 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824711 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824713 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824716 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824718 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824721 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824724 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824727 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.829303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824730 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.824735 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824841 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824846 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824849 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824852 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824855 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824858 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824861 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824864 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824866 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824869 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824871 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824874 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 04:48:54.829871 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824877 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824879 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824882 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824885 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824887 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824890 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824892 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824895 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824898 2560 feature_gate.go:328] unrecognized feature gate: Example Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824901 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824903 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824906 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824910 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824914 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824917 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824920 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824923 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824926 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824929 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824931 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 04:48:54.830219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824934 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824937 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824940 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824942 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824945 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824947 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824950 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824952 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824955 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824957 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824960 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824963 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824965 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824968 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824970 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824973 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824975 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824978 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824981 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824984 2560 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824986 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 04:48:54.830714 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824989 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824992 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824995 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.824998 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825000 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825003 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825005 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825008 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825011 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825013 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825016 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825019 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825021 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825024 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825026 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825029 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825031 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825034 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825037 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825039 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 04:48:54.831214 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825042 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825044 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825047 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825049 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825052 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825054 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825057 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825059 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825062 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825065 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825067 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825070 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825072 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825075 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825077 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825081 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825084 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825087 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825090 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 04:48:54.831728 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825092 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825095 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825098 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825102 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825106 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825109 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:54.825112 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.825117 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.825210 2560 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.828453 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.829577 2560 server.go:1019] "Starting client certificate rotation" Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.829682 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 04:48:54.832180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.829717 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 04:48:54.832513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.832374 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 04:48:54.833595 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.833577 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 04:48:54.844343 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.844324 2560 log.go:25] "Validated CRI v1 runtime API" Sep 16 04:48:54.848184 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.848169 2560 log.go:25] "Validated CRI v1 image API" Sep 16 04:48:54.848957 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.848938 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 04:48:54.851492 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.851473 2560 fs.go:135] Filesystem UUIDs: map[25fa5e50-f5ce-485f-9bea-50b09f9c4df2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f4a115ce-8577-4db3-9f7f-27897976e1f1:/dev/nvme0n1p4] Sep 16 04:48:54.851565 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.851491 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 04:48:54.857842 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.857717 2560 manager.go:216] Machine: {Timestamp:2026-09-16 04:48:54.856266504 +0000 UTC m=+0.369818029 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093102 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec280345f8a1f26835d64e6efadfacd6 SystemUUID:ec280345-f8a1-f268-35d6-4e6efadfacd6 BootID:a3d46d61-14e4-4d12-a098-ec80c2969a58 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:01:3f:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:01:3f:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:3b:21:a4:1a:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 04:48:54.857842 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.857843 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 04:48:54.857968 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.857956 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 04:48:54.861475 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.861452 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 04:48:54.861615 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.861479 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-144.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 04:48:54.862380 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.862371 2560 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 04:48:54.862412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.862383 2560 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 04:48:54.862412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.862391 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 04:48:54.863247 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.863233 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 04:48:54.863285 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.863264 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 04:48:54.864527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.864517 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 04:48:54.869177 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869166 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 16 04:48:54.869285 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869276 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 04:48:54.869317 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869307 2560 kubelet.go:486] "Attempting to sync node with API server" Sep 16 04:48:54.869345 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869322 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 04:48:54.869375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869345 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 04:48:54.869375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869358 2560 kubelet.go:398] "Adding apiserver pod source" Sep 16 04:48:54.869428 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869379 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 04:48:54.869769 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869758 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 04:48:54.869814 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.869775 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 04:48:54.872032 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.872010 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 04:48:54.874925 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.874911 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 04:48:54.874969 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.874941 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 04:48:54.877041 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877023 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877053 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877065 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877078 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877094 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877106 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877130 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 04:48:54.877148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877142 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 04:48:54.877331 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877153 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 04:48:54.877331 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877164 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 04:48:54.877331 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877190 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 04:48:54.877331 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.877201 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 04:48:54.878072 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.878062 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 04:48:54.878072 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.878072 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 04:48:54.881135 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.881116 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-144.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 04:48:54.881184 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.881167 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 04:48:54.881268 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.881239 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 04:48:54.882040 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.882030 2560 server.go:1268] "Started kubelet" Sep 16 04:48:54.882120 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.882098 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 04:48:54.882169 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.882115 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 04:48:54.882214 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.882203 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 04:48:54.883011 ip-10-0-142-144 systemd[1]: Started Kubernetes Kubelet. Sep 16 04:48:54.888282 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.888258 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 04:48:54.889274 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.889251 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 04:48:54.890461 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.890428 2560 server.go:310] "Adding debug handlers to kubelet server" Sep 16 04:48:54.893787 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.892732 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-144.ec2.internal.18d5b3d07713f075 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-144.ec2.internal,UID:ip-10-0-142-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-144.ec2.internal,},FirstTimestamp:2026-09-16 04:48:54.882005109 +0000 UTC m=+0.395556635,LastTimestamp:2026-09-16 04:48:54.882005109 +0000 UTC m=+0.395556635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-144.ec2.internal,}" Sep 16 04:48:54.894977 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.894963 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 04:48:54.895034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.894976 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 04:48:54.895091 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.895075 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:54.895748 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895735 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 04:48:54.895846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895756 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 04:48:54.895846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895811 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 04:48:54.895846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895771 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 04:48:54.895987 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895850 2560 factory.go:55] Registering systemd factory Sep 16 04:48:54.895987 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895860 2560 factory.go:223] Registration of the systemd container factory successfully Sep 16 04:48:54.895987 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895923 2560 reconstruct.go:97] "Volume reconstruction finished" Sep 16 04:48:54.895987 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.895934 2560 reconciler.go:29] "Reconciler: start to sync state" Sep 16 04:48:54.896123 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.896080 2560 factory.go:153] Registering CRI-O factory Sep 16 04:48:54.896123 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.896093 2560 factory.go:223] Registration of the crio container factory successfully Sep 16 04:48:54.896123 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.896111 2560 factory.go:103] Registering Raw factory Sep 16 04:48:54.896212 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.896142 2560 manager.go:1196] Started watching for new ooms in manager Sep 16 04:48:54.896529 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.896515 2560 manager.go:349] Starting recovery of all containers Sep 16 04:48:54.899983 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.899950 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-144.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 04:48:54.900097 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.900075 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 04:48:54.905078 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.905055 2560 manager.go:354] Recovery completed Sep 16 04:48:54.910327 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.910312 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:54.912669 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.912429 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:54.912669 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.912489 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:54.912669 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.912507 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:54.912669 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.912475 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4k672" Sep 16 04:48:54.913474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.913455 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 04:48:54.913665 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.913649 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 04:48:54.913723 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.913676 2560 state_mem.go:36] "Initialized new in-memory state store" Sep 16 04:48:54.913787 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.913538 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:54.915291 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.915273 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:54.915357 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.915307 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:54.915357 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.915321 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:54.915948 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.915882 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-144.ec2.internal.18d5b3d078e4c03b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-144.ec2.internal,UID:ip-10-0-142-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-144.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-144.ec2.internal,},FirstTimestamp:2026-09-16 04:48:54.912467003 +0000 UTC m=+0.426018532,LastTimestamp:2026-09-16 04:48:54.912467003 +0000 UTC m=+0.426018532,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-144.ec2.internal,}" Sep 16 04:48:54.918843 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:54.918826 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4k672" Sep 16 04:48:54.924689 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.924631 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-144.ec2.internal.18d5b3d078e5349c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-144.ec2.internal,UID:ip-10-0-142-144.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-144.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-144.ec2.internal,},FirstTimestamp:2026-09-16 04:48:54.912496796 +0000 UTC m=+0.426048321,LastTimestamp:2026-09-16 04:48:54.912496796 +0000 UTC m=+0.426048321,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-144.ec2.internal,}" Sep 16 04:48:54.995318 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:54.995291 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.056286 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.056261 2560 policy_none.go:49] "None policy: Start" Sep 16 04:48:55.056286 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.056285 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 04:48:55.056416 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.056297 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 04:48:55.058519 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.058453 2560 policy_none.go:47] "Start" Sep 16 04:48:55.092082 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092070 2560 manager.go:337] "Starting Device Plugin manager" Sep 16 04:48:55.092196 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.092105 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 04:48:55.092196 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092119 2560 server.go:87] "Starting device plugin registration server" Sep 16 04:48:55.092458 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092444 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 04:48:55.092509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092460 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 04:48:55.092581 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092564 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 04:48:55.092650 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092635 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 04:48:55.092650 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.092647 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 04:48:55.093211 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.093188 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 04:48:55.093307 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.093255 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.106293 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.106273 2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.113192 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.113167 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 04:48:55.114245 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.114224 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 04:48:55.114303 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.114272 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 04:48:55.114303 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.114294 2560 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 04:48:55.114379 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.114337 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 04:48:55.116906 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.116892 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:55.193359 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.193325 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:55.194321 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.194304 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:55.194413 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.194335 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:55.194413 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.194345 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:55.194413 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.194366 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.201491 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.201472 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.201617 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.201498 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-144.ec2.internal\": node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.214889 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.214867 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal"] Sep 16 04:48:55.214974 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.214928 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:55.216101 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.216087 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:55.216189 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.216119 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:55.216189 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.216137 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:55.218448 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.218433 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:55.218570 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.218538 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.218648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.218583 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:55.219120 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219096 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:55.219209 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219129 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:55.219209 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219144 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:55.219209 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219145 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:55.219334 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219218 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:55.219334 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.219229 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:55.221470 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.221456 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.221528 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.221481 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 04:48:55.222122 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.222105 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientMemory" Sep 16 04:48:55.222258 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.222133 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 04:48:55.222258 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.222147 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeHasSufficientPID" Sep 16 04:48:55.234370 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.234353 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.248227 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.248205 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.252446 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.252431 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-144.ec2.internal\" not found" node="ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.297796 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.297777 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.297867 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.297802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.297867 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.297817 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3871b89cd4ee6045bc4279bdebdf12fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"3871b89cd4ee6045bc4279bdebdf12fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.334436 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.334415 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.397922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.397887 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.397922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.397925 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.398032 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.397942 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3871b89cd4ee6045bc4279bdebdf12fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"3871b89cd4ee6045bc4279bdebdf12fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.398032 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.398009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.398096 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.398009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b3816fb36f563fd12a3625e83249ee5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal\" (UID: \"2b3816fb36f563fd12a3625e83249ee5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.398096 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.398003 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3871b89cd4ee6045bc4279bdebdf12fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-144.ec2.internal\" (UID: \"3871b89cd4ee6045bc4279bdebdf12fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.435023 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.435001 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.535499 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.535472 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.550651 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.550623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.555666 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.555424 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.636457 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.636392 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.737600 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.737575 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-144.ec2.internal\" not found" Sep 16 04:48:55.756066 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.756048 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:55.797465 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.797442 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.812829 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.812811 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 04:48:55.814229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.814215 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" Sep 16 04:48:55.827487 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.827472 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 04:48:55.829815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.829802 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 04:48:55.829917 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.829899 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 04:48:55.829961 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.829899 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 04:48:55.870529 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.870491 2560 apiserver.go:52] "Watching apiserver" Sep 16 04:48:55.888524 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.888455 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 04:48:55.890975 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.890948 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt","openshift-cluster-node-tuning-operator/tuned-rd7dd","openshift-image-registry/node-ca-vl57t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal","openshift-multus/multus-cz7c2","openshift-network-diagnostics/network-check-target-v92kt","kube-system/konnectivity-agent-9d9xk","openshift-dns/node-resolver-fx8ph","openshift-multus/multus-additional-cni-plugins-d4rtn","openshift-multus/network-metrics-daemon-gknpb","openshift-network-operator/iptables-alerter-f8pk6","openshift-ovn-kubernetes/ovnkube-node-r2bcd","kube-system/global-pull-secret-syncer-4jhmn","kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal"] Sep 16 04:48:55.893640 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.893625 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:55.893726 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.893709 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:48:55.895815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.895801 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 04:48:55.895946 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.895928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:55.898192 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.898177 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:55.898285 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.898240 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:48:55.899022 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.899000 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qprn5\"" Sep 16 04:48:55.900025 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.899998 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 04:48:55.900124 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900094 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:55.900189 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900135 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 04:48:55.900189 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5a7b348-8687-44f7-9e58-411bb466c01e-agent-certs\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:55.900290 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5a7b348-8687-44f7-9e58-411bb466c01e-konnectivity-ca\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:55.900290 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900272 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhb7r\" (UniqueName: \"kubernetes.io/projected/70065c80-52d4-4eb5-a606-905ee57917a9-kube-api-access-qhb7r\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:55.900390 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-kubelet-config\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:55.900390 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900342 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-dbus\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:55.900390 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.900368 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:55.901725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.901702 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz7c2" Sep 16 04:48:55.903925 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.903910 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:55.904008 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:55.903967 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:48:55.904350 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.904331 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.904432 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.904351 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 04:48:55.904432 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.904415 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 04:48:55.904950 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.904934 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4gxvp\"" Sep 16 04:48:55.905191 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.905176 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.906320 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.906304 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:55.908697 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.908681 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:55.909230 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.909214 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 04:48:55.909588 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.909558 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 04:48:55.909652 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.909613 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hg57j\"" Sep 16 04:48:55.910940 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.910924 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:55.911273 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.911256 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 04:48:55.911477 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.911460 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6w2sw\"" Sep 16 04:48:55.911556 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.911540 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.911867 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.911852 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.913397 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.913376 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:55.913511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.913490 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lpz4r\"" Sep 16 04:48:55.913883 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.913867 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.914088 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.914069 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.915913 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.915872 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:55.916281 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.916260 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 04:48:55.916738 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.916378 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.918433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.918411 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:55.920760 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.920740 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 04:48:55.920978 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.920959 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:55.921034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.921013 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.923031 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923009 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-txr2g\"" Sep 16 04:48:55.923146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923020 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 04:48:55.923381 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923363 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.923432 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923400 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.923474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923430 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lxrh7\"" Sep 16 04:48:55.923474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923367 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 04:48:55.923653 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923634 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.923703 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.923677 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 04:48:55.924260 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924247 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.924309 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924264 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-46qtf\"" Sep 16 04:48:55.924609 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924587 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 04:43:55 +0000 UTC" deadline="2028-05-22 04:08:35.150008795 +0000 UTC" Sep 16 04:48:55.924714 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924608 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14735h19m39.225403131s" Sep 16 04:48:55.924774 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924727 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rwdhj\"" Sep 16 04:48:55.924774 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924772 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 04:48:55.924892 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924792 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 04:48:55.924966 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.924917 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 04:48:55.925544 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.925531 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 04:48:55.950667 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.950648 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgmb8" Sep 16 04:48:55.959071 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.959053 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hgmb8" Sep 16 04:48:55.996814 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:55.996793 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 04:48:56.000661 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000641 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:56.000767 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000668 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-modprobe-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.000767 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.000763 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:56.000843 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.000820 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:48:56.500803834 +0000 UTC m=+2.014355348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:56.000903 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-socket-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.000903 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000864 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-etc-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.000903 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000889 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-config\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001049 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000914 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-kubernetes\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001049 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000936 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-netd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001049 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000958 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-cnibin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001049 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.000982 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-etc-kubernetes\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001049 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.001221 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001062 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-slash\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001221 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-ovn\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001221 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001179 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhb7r\" (UniqueName: \"kubernetes.io/projected/70065c80-52d4-4eb5-a606-905ee57917a9-kube-api-access-qhb7r\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:56.001221 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-dbus\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001222 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cqtp\" (UniqueName: \"kubernetes.io/projected/bee70a18-12a5-4290-952b-abc32375aa46-kube-api-access-7cqtp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001237 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-node-log\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001255 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-dbus\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001362 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5a7b348-8687-44f7-9e58-411bb466c01e-agent-certs\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-conf\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001395 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-sys\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001410 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-var-lib-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5sw\" (UniqueName: \"kubernetes.io/projected/695af7a0-6dde-4d7d-b185-32ef2b342741-kube-api-access-dk5sw\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001462 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-k8s-cni-cncf-io\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001482 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-daemon-config\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001497 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qghc\" (UniqueName: \"kubernetes.io/projected/8523d131-96e7-48c9-bc8c-4a99a787d299-kube-api-access-4qghc\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001511 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001529 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-bin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001552 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-systemd\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-var-lib-kubelet\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001611 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-sys-fs\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001631 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-netns\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001655 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-iptables-alerter-script\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001678 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001693 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-cnibin\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.001722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001700 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqbf\" (UniqueName: \"kubernetes.io/projected/7439855c-a586-4e18-b668-31e119086964-kube-api-access-njqbf\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001769 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-serviceca\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001819 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-lib-modules\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-device-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.001904 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001888 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovn-node-metrics-cert\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.001956 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:48:56.501940719 +0000 UTC m=+2.015492253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvh58\" (UniqueName: \"kubernetes.io/projected/6ed9bf57-c12e-499c-b4e3-abfd955aa783-kube-api-access-kvh58\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.001993 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002015 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-conf-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5a7b348-8687-44f7-9e58-411bb466c01e-konnectivity-ca\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002084 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np9jf\" (UniqueName: \"kubernetes.io/projected/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-kube-api-access-np9jf\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002102 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-bin\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.002146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002117 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-kubelet\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002133 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-hostroot\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002147 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-multus-certs\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002183 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002223 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695af7a0-6dde-4d7d-b185-32ef2b342741-hosts-file\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002251 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695af7a0-6dde-4d7d-b185-32ef2b342741-tmp-dir\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002276 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-system-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002302 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-multus\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002325 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-host\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002347 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-run\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkqj\" (UniqueName: \"kubernetes.io/projected/41032cf2-49bd-4075-b713-8890bc4dadb0-kube-api-access-9dkqj\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002397 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-systemd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002422 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-env-overrides\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002446 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-netns\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002470 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hc5w\" (UniqueName: \"kubernetes.io/projected/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-kube-api-access-4hc5w\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002519 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-host\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.002860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002543 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-host-slash\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002553 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5a7b348-8687-44f7-9e58-411bb466c01e-konnectivity-ca\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-kubelet\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002640 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-socket-dir-parent\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysconfig\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002707 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-etc-tuned\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-registration-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002741 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-systemd-units\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-script-lib\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002782 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-system-cni-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002797 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-kubelet-config\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-tmp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002856 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002878 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-log-socket\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003459 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002899 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.003948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-cni-binary-copy\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.003948 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.002920 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/20993e50-9419-404f-a4b6-0eda4a2afeff-kubelet-config\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.005653 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.005619 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5a7b348-8687-44f7-9e58-411bb466c01e-agent-certs\") pod \"konnectivity-agent-9d9xk\" (UID: \"b5a7b348-8687-44f7-9e58-411bb466c01e\") " pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:56.010206 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.010188 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhb7r\" (UniqueName: \"kubernetes.io/projected/70065c80-52d4-4eb5-a606-905ee57917a9-kube-api-access-qhb7r\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:56.103409 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103245 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-systemd\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.103409 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103331 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-systemd\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103439 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-var-lib-kubelet\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103462 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-sys-fs\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103478 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-netns\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-iptables-alerter-script\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103550 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-var-lib-kubelet\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-sys-fs\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.103687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103572 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-netns\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103738 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-cnibin\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqbf\" (UniqueName: \"kubernetes.io/projected/7439855c-a586-4e18-b668-31e119086964-kube-api-access-njqbf\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-serviceca\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103829 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-lib-modules\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103841 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-cnibin\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103929 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103942 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-lib-modules\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.103967 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-device-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104011 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovn-node-metrics-cert\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvh58\" (UniqueName: \"kubernetes.io/projected/6ed9bf57-c12e-499c-b4e3-abfd955aa783-kube-api-access-kvh58\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.104069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104069 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-device-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104105 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-iptables-alerter-script\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104176 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-conf-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104218 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np9jf\" (UniqueName: \"kubernetes.io/projected/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-kube-api-access-np9jf\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104236 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-bin\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-kubelet\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104253 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-conf-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-hostroot\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104291 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-serviceca\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104307 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-bin\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-kubelet\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104300 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-multus-certs\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104359 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-hostroot\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104359 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104361 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-multus-certs\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695af7a0-6dde-4d7d-b185-32ef2b342741-hosts-file\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.104813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104426 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695af7a0-6dde-4d7d-b185-32ef2b342741-tmp-dir\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104451 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-system-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104473 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/695af7a0-6dde-4d7d-b185-32ef2b342741-hosts-file\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104477 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-multus\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104523 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-host\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-run\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104591 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkqj\" (UniqueName: \"kubernetes.io/projected/41032cf2-49bd-4075-b713-8890bc4dadb0-kube-api-access-9dkqj\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-system-cni-dir\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104621 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-systemd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104627 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-host\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-env-overrides\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-run\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104688 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-netns\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104729 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/695af7a0-6dde-4d7d-b185-32ef2b342741-tmp-dir\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104691 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-multus\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104729 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104744 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-systemd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104784 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hc5w\" (UniqueName: \"kubernetes.io/projected/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-kube-api-access-4hc5w\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.105877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-netns\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-host\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104938 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-host-slash\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104963 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-binary-copy\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.104999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-kubelet\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105017 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-host-slash\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105026 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105058 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-socket-dir-parent\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105099 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-kubelet\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105113 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysconfig\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-etc-tuned\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105149 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-host\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105167 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-env-overrides\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105173 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-registration-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105185 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-socket-dir-parent\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-systemd-units\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.106833 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysconfig\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105241 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-script-lib\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105268 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-registration-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105267 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-system-cni-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105321 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-systemd-units\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105323 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-system-cni-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-tmp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105507 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7439855c-a586-4e18-b668-31e119086964-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105551 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-log-socket\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105662 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-etc-selinux\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105674 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-log-socket\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105705 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-cni-binary-copy\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105745 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-modprobe-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.107935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105704 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105767 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-socket-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105794 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-etc-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-config\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105863 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-modprobe-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105922 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-kubernetes\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-kubernetes\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-netd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105978 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/41032cf2-49bd-4075-b713-8890bc4dadb0-socket-dir\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.105992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-cnibin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106042 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-etc-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106095 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-cnibin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-cni-netd\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106167 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-etc-kubernetes\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106196 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106220 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-cni-binary-copy\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-slash\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-slash\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.108932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106280 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-ovn\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106299 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-etc-kubernetes\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106311 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cqtp\" (UniqueName: \"kubernetes.io/projected/bee70a18-12a5-4290-952b-abc32375aa46-kube-api-access-7cqtp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106337 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-node-log\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106388 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106417 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-conf\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-sys\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-var-lib-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5sw\" (UniqueName: \"kubernetes.io/projected/695af7a0-6dde-4d7d-b185-32ef2b342741-kube-api-access-dk5sw\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106519 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-k8s-cni-cncf-io\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-daemon-config\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106545 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-config\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106561 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovnkube-script-lib\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106567 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qghc\" (UniqueName: \"kubernetes.io/projected/8523d131-96e7-48c9-bc8c-4a99a787d299-kube-api-access-4qghc\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106613 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106683 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-bin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106773 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-var-lib-cni-bin\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106817 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-conf\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.109988 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106824 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-sys\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106841 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7439855c-a586-4e18-b668-31e119086964-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106857 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-var-lib-openvswitch\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106879 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-run-ovn\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106883 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bee70a18-12a5-4290-952b-abc32375aa46-etc-sysctl-d\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ed9bf57-c12e-499c-b4e3-abfd955aa783-ovn-node-metrics-cert\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106927 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8523d131-96e7-48c9-bc8c-4a99a787d299-host-run-k8s-cni-cncf-io\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106951 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-node-log\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.106971 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6ed9bf57-c12e-499c-b4e3-abfd955aa783-host-run-ovn-kubernetes\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.107357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8523d131-96e7-48c9-bc8c-4a99a787d299-multus-daemon-config\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.107703 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-etc-tuned\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.110898 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.108536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bee70a18-12a5-4290-952b-abc32375aa46-tmp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.111324 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.111307 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:48:56.111361 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.111328 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:48:56.111361 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.111338 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:56.111431 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.111386 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:48:56.611373187 +0000 UTC m=+2.124924701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:56.113179 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.113151 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np9jf\" (UniqueName: \"kubernetes.io/projected/22b6fa96-0c2c-4140-9f2a-bd7a404a58b4-kube-api-access-np9jf\") pod \"iptables-alerter-f8pk6\" (UID: \"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4\") " pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.114391 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.114368 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvh58\" (UniqueName: \"kubernetes.io/projected/6ed9bf57-c12e-499c-b4e3-abfd955aa783-kube-api-access-kvh58\") pod \"ovnkube-node-r2bcd\" (UID: \"6ed9bf57-c12e-499c-b4e3-abfd955aa783\") " pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.114632 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.114607 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qghc\" (UniqueName: \"kubernetes.io/projected/8523d131-96e7-48c9-bc8c-4a99a787d299-kube-api-access-4qghc\") pod \"multus-cz7c2\" (UID: \"8523d131-96e7-48c9-bc8c-4a99a787d299\") " pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.114697 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.114686 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkqj\" (UniqueName: \"kubernetes.io/projected/41032cf2-49bd-4075-b713-8890bc4dadb0-kube-api-access-9dkqj\") pod \"aws-ebs-csi-driver-node-8gfzt\" (UID: \"41032cf2-49bd-4075-b713-8890bc4dadb0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.114929 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.114913 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqbf\" (UniqueName: \"kubernetes.io/projected/7439855c-a586-4e18-b668-31e119086964-kube-api-access-njqbf\") pod \"multus-additional-cni-plugins-d4rtn\" (UID: \"7439855c-a586-4e18-b668-31e119086964\") " pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.115010 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.114989 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hc5w\" (UniqueName: \"kubernetes.io/projected/6155ff74-4c91-44f2-8bd7-6768c8fe3de8-kube-api-access-4hc5w\") pod \"node-ca-vl57t\" (UID: \"6155ff74-4c91-44f2-8bd7-6768c8fe3de8\") " pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.115684 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.115666 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cqtp\" (UniqueName: \"kubernetes.io/projected/bee70a18-12a5-4290-952b-abc32375aa46-kube-api-access-7cqtp\") pod \"tuned-rd7dd\" (UID: \"bee70a18-12a5-4290-952b-abc32375aa46\") " pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.118653 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.118615 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5sw\" (UniqueName: \"kubernetes.io/projected/695af7a0-6dde-4d7d-b185-32ef2b342741-kube-api-access-dk5sw\") pod \"node-resolver-fx8ph\" (UID: \"695af7a0-6dde-4d7d-b185-32ef2b342741\") " pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.221312 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.221245 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:48:56.238517 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.238486 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cz7c2" Sep 16 04:48:56.255365 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.255341 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" Sep 16 04:48:56.256831 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.256802 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3871b89cd4ee6045bc4279bdebdf12fc.slice/crio-36d86434e20a1d59d71d4fa2f718b9957833b2fdd051755a4b54f0e3dcfbb648 WatchSource:0}: Error finding container 36d86434e20a1d59d71d4fa2f718b9957833b2fdd051755a4b54f0e3dcfbb648: Status 404 returned error can't find the container with id 36d86434e20a1d59d71d4fa2f718b9957833b2fdd051755a4b54f0e3dcfbb648 Sep 16 04:48:56.257050 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.257030 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a7b348_8687_44f7_9e58_411bb466c01e.slice/crio-96f15cf0b1cc5cb1d91d480ef91d118e4a9e7759ecf902821805473655e03d37 WatchSource:0}: Error finding container 96f15cf0b1cc5cb1d91d480ef91d118e4a9e7759ecf902821805473655e03d37: Status 404 returned error can't find the container with id 96f15cf0b1cc5cb1d91d480ef91d118e4a9e7759ecf902821805473655e03d37 Sep 16 04:48:56.258183 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.258156 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vl57t" Sep 16 04:48:56.261991 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.261921 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3816fb36f563fd12a3625e83249ee5.slice/crio-2e683775b5d16f85861fed16a8b5d3bd64bf8fed5ab31f60faa6c25d02536e51 WatchSource:0}: Error finding container 2e683775b5d16f85861fed16a8b5d3bd64bf8fed5ab31f60faa6c25d02536e51: Status 404 returned error can't find the container with id 2e683775b5d16f85861fed16a8b5d3bd64bf8fed5ab31f60faa6c25d02536e51 Sep 16 04:48:56.262906 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.262889 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8523d131_96e7_48c9_bc8c_4a99a787d299.slice/crio-78b4efee6086ab2903b80404edf7840368a5f306e0c0fae063b76e265b1e9747 WatchSource:0}: Error finding container 78b4efee6086ab2903b80404edf7840368a5f306e0c0fae063b76e265b1e9747: Status 404 returned error can't find the container with id 78b4efee6086ab2903b80404edf7840368a5f306e0c0fae063b76e265b1e9747 Sep 16 04:48:56.264569 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.264548 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 04:48:56.271392 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.271370 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7439855c_a586_4e18_b668_31e119086964.slice/crio-9bc77d3c98f77ce06bf63638f3df2aba3e6898abf491671647f77bf280b0ae9e WatchSource:0}: Error finding container 9bc77d3c98f77ce06bf63638f3df2aba3e6898abf491671647f77bf280b0ae9e: Status 404 returned error can't find the container with id 9bc77d3c98f77ce06bf63638f3df2aba3e6898abf491671647f77bf280b0ae9e Sep 16 04:48:56.272067 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.272046 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155ff74_4c91_44f2_8bd7_6768c8fe3de8.slice/crio-c5f855c4ca2a75e736d541827d0a089171ae9678a809c86adc3ecca395fa1211 WatchSource:0}: Error finding container c5f855c4ca2a75e736d541827d0a089171ae9678a809c86adc3ecca395fa1211: Status 404 returned error can't find the container with id c5f855c4ca2a75e736d541827d0a089171ae9678a809c86adc3ecca395fa1211 Sep 16 04:48:56.273298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.273277 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" Sep 16 04:48:56.281595 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.281569 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" Sep 16 04:48:56.282325 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.282302 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee70a18_12a5_4290_952b_abc32375aa46.slice/crio-833f5817cc4ba93482d598ddd61c16e6708181455fd0b938023f1a61890305c1 WatchSource:0}: Error finding container 833f5817cc4ba93482d598ddd61c16e6708181455fd0b938023f1a61890305c1: Status 404 returned error can't find the container with id 833f5817cc4ba93482d598ddd61c16e6708181455fd0b938023f1a61890305c1 Sep 16 04:48:56.290651 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.290626 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41032cf2_49bd_4075_b713_8890bc4dadb0.slice/crio-6a78d3c2b977c6716caa0d9ae80bb6fe170634426c147508a68ec5ab63880872 WatchSource:0}: Error finding container 6a78d3c2b977c6716caa0d9ae80bb6fe170634426c147508a68ec5ab63880872: Status 404 returned error can't find the container with id 6a78d3c2b977c6716caa0d9ae80bb6fe170634426c147508a68ec5ab63880872 Sep 16 04:48:56.292710 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.292696 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8pk6" Sep 16 04:48:56.292823 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.292806 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:56.298682 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.298663 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:48:56.301182 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.301164 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b6fa96_0c2c_4140_9f2a_bd7a404a58b4.slice/crio-b6cbf7d6dc1baa419eb70e6820a8340b4a49033764f88327ad78a93b9d2293e7 WatchSource:0}: Error finding container b6cbf7d6dc1baa419eb70e6820a8340b4a49033764f88327ad78a93b9d2293e7: Status 404 returned error can't find the container with id b6cbf7d6dc1baa419eb70e6820a8340b4a49033764f88327ad78a93b9d2293e7 Sep 16 04:48:56.307610 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.307585 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed9bf57_c12e_499c_b4e3_abfd955aa783.slice/crio-76b3fc0974af2d94add77708a15d6e30303b3a8d54e0736f253797fa9f907fb6 WatchSource:0}: Error finding container 76b3fc0974af2d94add77708a15d6e30303b3a8d54e0736f253797fa9f907fb6: Status 404 returned error can't find the container with id 76b3fc0974af2d94add77708a15d6e30303b3a8d54e0736f253797fa9f907fb6 Sep 16 04:48:56.314452 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.314435 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fx8ph" Sep 16 04:48:56.323945 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:48:56.323927 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695af7a0_6dde_4d7d_b185_32ef2b342741.slice/crio-913aa8192a6b46e8a31db44d4f39af4e78e6d3357be56350cf56851fbe68eb56 WatchSource:0}: Error finding container 913aa8192a6b46e8a31db44d4f39af4e78e6d3357be56350cf56851fbe68eb56: Status 404 returned error can't find the container with id 913aa8192a6b46e8a31db44d4f39af4e78e6d3357be56350cf56851fbe68eb56 Sep 16 04:48:56.325449 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.325431 2560 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-fx8ph_openshift-dns(695af7a0-6dde-4d7d-b185-32ef2b342741): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 16 04:48:56.325512 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.325462 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-fx8ph" podUID="695af7a0-6dde-4d7d-b185-32ef2b342741" Sep 16 04:48:56.412639 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.412609 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:56.509690 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.509618 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:56.509690 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.509666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:56.509859 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.509770 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:56.509859 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.509826 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:48:57.509811846 +0000 UTC m=+3.023363365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:56.509859 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.509845 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:56.509984 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.509885 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:48:57.509874543 +0000 UTC m=+3.023426056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:56.711151 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.711117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:56.711335 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.711275 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:48:56.711335 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.711305 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:48:56.711335 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.711318 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:56.711503 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:56.711369 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:48:57.711356668 +0000 UTC m=+3.224908186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:56.821865 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.821789 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:56.964275 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.962378 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 04:43:56 +0000 UTC" deadline="2028-05-30 07:11:10.57806361 +0000 UTC" Sep 16 04:48:56.964275 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:56.962435 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14930h22m13.615633316s" Sep 16 04:48:57.119305 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.119271 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:57.119485 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.119410 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:48:57.119881 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.119859 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:57.120003 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.119981 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:48:57.126834 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.126784 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" event={"ID":"41032cf2-49bd-4075-b713-8890bc4dadb0","Type":"ContainerStarted","Data":"6a78d3c2b977c6716caa0d9ae80bb6fe170634426c147508a68ec5ab63880872"} Sep 16 04:48:57.129330 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.129301 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vl57t" event={"ID":"6155ff74-4c91-44f2-8bd7-6768c8fe3de8","Type":"ContainerStarted","Data":"c5f855c4ca2a75e736d541827d0a089171ae9678a809c86adc3ecca395fa1211"} Sep 16 04:48:57.149141 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.149107 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerStarted","Data":"9bc77d3c98f77ce06bf63638f3df2aba3e6898abf491671647f77bf280b0ae9e"} Sep 16 04:48:57.179270 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.179229 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"2b3816fb36f563fd12a3625e83249ee5","Type":"ContainerStarted","Data":"2e683775b5d16f85861fed16a8b5d3bd64bf8fed5ab31f60faa6c25d02536e51"} Sep 16 04:48:57.190524 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.190407 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9d9xk" event={"ID":"b5a7b348-8687-44f7-9e58-411bb466c01e","Type":"ContainerStarted","Data":"96f15cf0b1cc5cb1d91d480ef91d118e4a9e7759ecf902821805473655e03d37"} Sep 16 04:48:57.203063 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.203038 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fx8ph" event={"ID":"695af7a0-6dde-4d7d-b185-32ef2b342741","Type":"ContainerStarted","Data":"913aa8192a6b46e8a31db44d4f39af4e78e6d3357be56350cf56851fbe68eb56"} Sep 16 04:48:57.208132 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.208102 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-fx8ph" podUID="695af7a0-6dde-4d7d-b185-32ef2b342741" Sep 16 04:48:57.211283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.211255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"76b3fc0974af2d94add77708a15d6e30303b3a8d54e0736f253797fa9f907fb6"} Sep 16 04:48:57.228923 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.228889 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8pk6" event={"ID":"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4","Type":"ContainerStarted","Data":"b6cbf7d6dc1baa419eb70e6820a8340b4a49033764f88327ad78a93b9d2293e7"} Sep 16 04:48:57.247660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.247611 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" event={"ID":"bee70a18-12a5-4290-952b-abc32375aa46","Type":"ContainerStarted","Data":"833f5817cc4ba93482d598ddd61c16e6708181455fd0b938023f1a61890305c1"} Sep 16 04:48:57.273452 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.273418 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz7c2" event={"ID":"8523d131-96e7-48c9-bc8c-4a99a787d299","Type":"ContainerStarted","Data":"78b4efee6086ab2903b80404edf7840368a5f306e0c0fae063b76e265b1e9747"} Sep 16 04:48:57.299651 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.299616 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" event={"ID":"3871b89cd4ee6045bc4279bdebdf12fc","Type":"ContainerStarted","Data":"36d86434e20a1d59d71d4fa2f718b9957833b2fdd051755a4b54f0e3dcfbb648"} Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.517319 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.517499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.517670 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.517751 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.517731952 +0000 UTC m=+5.031283467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.518196 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:57.518325 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.518245 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.518230396 +0000 UTC m=+5.031781929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:57.719471 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.719433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:57.719706 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.719687 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:48:57.719787 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.719717 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:48:57.719787 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.719731 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:57.719907 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:57.719793 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:48:59.71977232 +0000 UTC m=+5.233323857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:57.768729 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.768660 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 04:48:57.962763 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.962700 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 04:43:56 +0000 UTC" deadline="2028-06-12 17:59:11.17058624 +0000 UTC" Sep 16 04:48:57.962763 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:57.962761 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15253h10m13.207829359s" Sep 16 04:48:58.115090 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:58.115009 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:58.115313 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:58.115148 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:48:59.116125 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:59.115615 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:59.116125 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.115757 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:48:59.116125 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:59.116107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:59.116761 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.116226 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:59.535400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:59.535481 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.535623 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.535690 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:49:03.535671294 +0000 UTC m=+9.049222822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.536084 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:59.536696 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.536133 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:03.536116932 +0000 UTC m=+9.049668461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:48:59.737665 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:48:59.737004 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:48:59.737665 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.737224 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:48:59.737665 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.737246 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:48:59.737665 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.737260 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:48:59.737665 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:48:59.737318 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:03.737299537 +0000 UTC m=+9.250851056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:00.114804 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:00.114765 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:00.115004 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:00.114928 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:01.115702 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:01.115665 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:01.116199 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:01.115814 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:01.116335 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:01.116313 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:01.116458 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:01.116431 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:02.115315 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:02.115277 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:02.115477 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:02.115411 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:03.115635 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:03.115593 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:03.116131 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.115747 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:03.116131 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:03.115799 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:03.116131 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.115955 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:03.572512 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:03.572463 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:03.572706 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:03.572559 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:03.572706 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.572646 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:03.572706 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.572684 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:03.572862 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.572720 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:49:11.572699571 +0000 UTC m=+17.086251100 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:03.572862 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.572752 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:11.572732971 +0000 UTC m=+17.086284495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:03.774077 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:03.773992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:03.774272 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.774248 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:49:03.774326 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.774284 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:49:03.774326 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.774299 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:03.774423 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:03.774366 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:11.774346313 +0000 UTC m=+17.287897841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:04.114562 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:04.114525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:04.114758 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:04.114679 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:05.116578 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:05.116003 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:05.116578 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:05.116127 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:05.116578 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:05.116434 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:05.116578 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:05.116520 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:06.115070 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:06.115027 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:06.115331 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:06.115159 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:07.115364 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:07.115319 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:07.115832 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:07.115459 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:07.115832 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:07.115504 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:07.115832 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:07.115605 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:08.114619 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:08.114525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:08.114767 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:08.114659 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:09.115521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:09.115482 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:09.115521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:09.115503 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:09.116004 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:09.115626 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:09.116004 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:09.115711 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:10.115376 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:10.115315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:10.115580 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:10.115458 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:11.114941 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:11.114857 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:11.115127 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.114990 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:11.115127 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:11.115040 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:11.115237 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.115137 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:11.633831 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:11.633779 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:11.634452 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:11.633850 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:11.634452 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.633964 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:11.634452 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.633993 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:11.634452 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.634033 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:49:27.634016898 +0000 UTC m=+33.147568415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:11.634452 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.634051 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:27.634042417 +0000 UTC m=+33.147593928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:11.835026 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:11.834992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:11.835208 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.835187 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:49:11.835259 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.835213 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:49:11.835259 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.835225 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:11.835341 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:11.835276 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:27.835262396 +0000 UTC m=+33.348813909 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:12.114777 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:12.114696 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:12.114944 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:12.114809 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:13.114597 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:13.114479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:13.115132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:13.114627 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:13.115132 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:13.114628 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:13.115132 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:13.114721 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:14.115215 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:14.115184 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:14.115712 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:14.115320 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:15.115177 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:15.115150 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:15.115352 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:15.115240 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:15.115352 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:15.115336 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:15.115780 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:15.115461 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:16.115320 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.115175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:16.115442 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:16.115423 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:16.349936 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.349918 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:49:16.350320 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.350294 2560 generic.go:356] "Generic (PLEG): container finished" podID="6ed9bf57-c12e-499c-b4e3-abfd955aa783" containerID="9a6a5d50eb2a3dcc5a82718fc0f716f4a6924d86a244f959724cb4a0e15348b4" exitCode=1 Sep 16 04:49:16.350404 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.350376 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"ab2a6935fb32a4e39776d6d23c4e5f2df94e0a996eed5e06d34b21cfe5cb64d8"} Sep 16 04:49:16.350475 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.350458 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"1bb3b9ec9c96b8b8560acc7c7e65c5dc130e7d1028ee0b8e570460eff73b15a0"} Sep 16 04:49:16.350532 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.350514 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerDied","Data":"9a6a5d50eb2a3dcc5a82718fc0f716f4a6924d86a244f959724cb4a0e15348b4"} Sep 16 04:49:16.350661 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.350582 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"33a5d152dfc1895d5261a300e9f40cacdbc14ea5422563ed4e8f961c692134b0"} Sep 16 04:49:16.352183 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.352161 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" event={"ID":"bee70a18-12a5-4290-952b-abc32375aa46","Type":"ContainerStarted","Data":"5fa455222dd1485205829fa93c7363ee18cac7b42748a1a6a33622f42cf03449"} Sep 16 04:49:16.353734 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.353710 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" event={"ID":"3871b89cd4ee6045bc4279bdebdf12fc","Type":"ContainerStarted","Data":"fcf505c0c71ff4cba55b8a0e38ed4815deff9c1ad89621b5702d21846fa14dc0"} Sep 16 04:49:16.369263 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.369210 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rd7dd" podStartSLOduration=0.980510148 podStartE2EDuration="20.36919633s" totalImagesPullingTime="19.388686182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:16.36858761 +0000 UTC m=+21.882139144" watchObservedRunningTime="2026-09-16 04:49:16.36919633 +0000 UTC m=+21.882747863" Sep 16 04:49:16.381198 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:16.381153 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-144.ec2.internal" podStartSLOduration=20.381134176 podStartE2EDuration="20.381134176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:16.380747477 +0000 UTC m=+21.894299014" watchObservedRunningTime="2026-09-16 04:49:16.381134176 +0000 UTC m=+21.894685712" Sep 16 04:49:17.114722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.114543 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:17.114909 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.114564 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:17.114909 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:17.114790 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:17.114909 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:17.114866 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:17.358817 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.358785 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:49:17.359588 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.359230 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"f77c5a6d7ce1e4d4a57dfe41f606507fce503c1df30d61e839ef5bb7beaac90e"} Sep 16 04:49:17.359588 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.359307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"78b03c41ee2a112297d2fc66670e790fc156e57cfd716aa57417b463a4a53337"} Sep 16 04:49:17.361048 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.361023 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8pk6" event={"ID":"22b6fa96-0c2c-4140-9f2a-bd7a404a58b4","Type":"ContainerStarted","Data":"55e11370da488ae1186bf096464c1a1b74f12890b0a8169183e8d048b2e81245"} Sep 16 04:49:17.362802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.362772 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cz7c2" event={"ID":"8523d131-96e7-48c9-bc8c-4a99a787d299","Type":"ContainerStarted","Data":"82e63b02031431aac78fa90e51aafaf19355c201fe3236bbe9cbe06c8b543726"} Sep 16 04:49:17.364354 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.364328 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" event={"ID":"41032cf2-49bd-4075-b713-8890bc4dadb0","Type":"ContainerStarted","Data":"415d4de1d9d7903b90389d0c861fb542c28f508eb97ddfa578d6309dabd07bd1"} Sep 16 04:49:17.366230 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.366172 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vl57t" event={"ID":"6155ff74-4c91-44f2-8bd7-6768c8fe3de8","Type":"ContainerStarted","Data":"c9a5309e1944f3d12b5a3dc6bae7ca93bca197b0a4aa058a56c9271854952a3a"} Sep 16 04:49:17.367740 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.367714 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="2ab74acd3496b1e57c5fe9b87a176ecba96181eeb1f5110eeacc3f25d6dc0717" exitCode=0 Sep 16 04:49:17.367846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.367788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"2ab74acd3496b1e57c5fe9b87a176ecba96181eeb1f5110eeacc3f25d6dc0717"} Sep 16 04:49:17.369648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.369628 2560 generic.go:356] "Generic (PLEG): container finished" podID="2b3816fb36f563fd12a3625e83249ee5" containerID="ebabc985467b1c6148158176026436072053077386bd0699854ae614fd6444a5" exitCode=0 Sep 16 04:49:17.369756 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.369658 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"2b3816fb36f563fd12a3625e83249ee5","Type":"ContainerDied","Data":"ebabc985467b1c6148158176026436072053077386bd0699854ae614fd6444a5"} Sep 16 04:49:17.371454 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.371431 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9d9xk" event={"ID":"b5a7b348-8687-44f7-9e58-411bb466c01e","Type":"ContainerStarted","Data":"b8db013fb9aa87dffdbd22fdda11c37ad70c8bb498e41a6aab40e7916ea3743d"} Sep 16 04:49:17.373046 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.373025 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fx8ph" event={"ID":"695af7a0-6dde-4d7d-b185-32ef2b342741","Type":"ContainerStarted","Data":"2e8a12529613ac7f3546654954401a45bea10396ce2e5a6908c0d0af8e32d9b7"} Sep 16 04:49:17.374919 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.374881 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f8pk6" podStartSLOduration=2.006157108 podStartE2EDuration="21.374866409s" totalImagesPullingTime="19.368709301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:17.374526466 +0000 UTC m=+22.888077992" watchObservedRunningTime="2026-09-16 04:49:17.374866409 +0000 UTC m=+22.888417943" Sep 16 04:49:17.388493 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.388457 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fx8ph" podStartSLOduration=21.388445199 podStartE2EDuration="21.388445199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-16 04:49:17.388336281 +0000 UTC m=+22.901887815" watchObservedRunningTime="2026-09-16 04:49:17.388445199 +0000 UTC m=+22.901996734" Sep 16 04:49:17.406676 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.406633 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cz7c2" podStartSLOduration=1.411748327 podStartE2EDuration="21.406617018s" totalImagesPullingTime="19.994868691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:17.406013099 +0000 UTC m=+22.919564669" watchObservedRunningTime="2026-09-16 04:49:17.406617018 +0000 UTC m=+22.920168554" Sep 16 04:49:17.438651 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.438597 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vl57t" podStartSLOduration=6.174631975 podStartE2EDuration="21.438577083s" totalImagesPullingTime="15.263945108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:17.43831548 +0000 UTC m=+22.951867018" watchObservedRunningTime="2026-09-16 04:49:17.438577083 +0000 UTC m=+22.952128615" Sep 16 04:49:17.468375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.468332 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9d9xk" podStartSLOduration=3.061894703 podStartE2EDuration="22.468316588s" totalImagesPullingTime="19.406421885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:17.467688906 +0000 UTC m=+22.981240441" watchObservedRunningTime="2026-09-16 04:49:17.468316588 +0000 UTC m=+22.981868122" Sep 16 04:49:17.936061 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:17.935922 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 04:49:18.103741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.103543 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T04:49:17.93605537Z","UUID":"8e2c4905-922f-476f-9c00-9029cb68a01b","Handler":null,"Name":"","Endpoint":""} Sep 16 04:49:18.105257 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.105231 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 04:49:18.105370 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.105265 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 04:49:18.114515 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.114494 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:18.114676 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:18.114605 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:18.377677 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.377584 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" event={"ID":"41032cf2-49bd-4075-b713-8890bc4dadb0","Type":"ContainerStarted","Data":"f60284ef7be2b4a0ad7205d871f68eab0939bfa3c724150491dc46956e12ebf2"} Sep 16 04:49:18.379688 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.379655 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" event={"ID":"2b3816fb36f563fd12a3625e83249ee5","Type":"ContainerStarted","Data":"ea59c14572402d619b7bc276dcc4dc40166cf37bb5b38b7aeb6d820ee95cfe3e"} Sep 16 04:49:18.395994 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:18.395947 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-144.ec2.internal" podStartSLOduration=22.395929224 podStartE2EDuration="22.395929224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:18.395229812 +0000 UTC m=+23.908781346" watchObservedRunningTime="2026-09-16 04:49:18.395929224 +0000 UTC m=+23.909480759" Sep 16 04:49:19.115039 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.115008 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:19.115196 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:19.115136 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:19.115265 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.115193 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:19.115354 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:19.115304 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:19.385810 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.385756 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:49:19.386344 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.386184 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"5b753eb2c4721d4189bfeede980032bc35a19c4f0f170f543fb63ad1dbf841fd"} Sep 16 04:49:19.388484 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.388460 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" event={"ID":"41032cf2-49bd-4075-b713-8890bc4dadb0","Type":"ContainerStarted","Data":"7de90e11d52d65ec601af894f542ef260a10af69e067ea4872a9a961f647232f"} Sep 16 04:49:19.409064 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:19.409020 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8gfzt" podStartSLOduration=2.272901636 podStartE2EDuration="23.409006271s" totalImagesPullingTime="21.136104635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:19.408452438 +0000 UTC m=+24.922003973" watchObservedRunningTime="2026-09-16 04:49:19.409006271 +0000 UTC m=+24.922557805" Sep 16 04:49:20.114489 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:20.114459 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:20.114652 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:20.114579 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:20.761211 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:20.761179 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:49:20.761859 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:20.761837 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:49:21.115233 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:21.115200 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:21.115411 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:21.115314 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:21.118731 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:21.115739 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:21.118731 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:21.115867 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:21.393983 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:21.393914 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:49:21.394873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:21.394854 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9d9xk" Sep 16 04:49:22.115081 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.114925 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:22.115431 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:22.115145 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:22.397879 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.397849 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="5e07475f44a74e5dadd12d2f1359cb92eb77971410c48cee453535fc83a23484" exitCode=0 Sep 16 04:49:22.398057 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.397942 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"5e07475f44a74e5dadd12d2f1359cb92eb77971410c48cee453535fc83a23484"} Sep 16 04:49:22.401637 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.401620 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:49:22.402026 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.402009 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"8a3b3200f54be0009afb909cab0e33bcea5f23e62854593e12658565ae5d4912"} Sep 16 04:49:22.402417 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:22.402402 2560 scope.go:117] "RemoveContainer" containerID="9a6a5d50eb2a3dcc5a82718fc0f716f4a6924d86a244f959724cb4a0e15348b4" Sep 16 04:49:23.115306 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.115272 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:23.116542 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.115288 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:23.116542 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:23.115407 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:23.116542 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:23.115522 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:23.409197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.409178 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:49:23.409621 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.409589 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" event={"ID":"6ed9bf57-c12e-499c-b4e3-abfd955aa783","Type":"ContainerStarted","Data":"bc4de57eb26fa15bc6b454f4f9467b90bde433c2acdded0a1bd1644b8d3a1b40"} Sep 16 04:49:23.410090 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.410061 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:23.410201 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.410155 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:23.410256 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.410207 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:23.412468 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.412441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerStarted","Data":"b9729aa90d6451a7459cabc5d6e38bfc7cdb7b01e117decbdb26ee44a922ba6e"} Sep 16 04:49:23.424721 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.424699 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:23.425105 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.425089 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:23.441620 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.441572 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" podStartSLOduration=8.064436096 podStartE2EDuration="27.441539441s" totalImagesPullingTime="19.377103345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:23.439618453 +0000 UTC m=+28.953169987" watchObservedRunningTime="2026-09-16 04:49:23.441539441 +0000 UTC m=+28.955090976" Sep 16 04:49:23.599205 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.599178 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gknpb"] Sep 16 04:49:23.599375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.599331 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:23.599453 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:23.599431 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:23.602424 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.602397 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v92kt"] Sep 16 04:49:23.602578 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.602489 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:23.602646 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:23.602589 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:23.603015 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.602983 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jhmn"] Sep 16 04:49:23.603090 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:23.603077 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:23.603186 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:23.603167 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:24.416381 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:24.416351 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="b9729aa90d6451a7459cabc5d6e38bfc7cdb7b01e117decbdb26ee44a922ba6e" exitCode=0 Sep 16 04:49:24.417075 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:24.416434 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"b9729aa90d6451a7459cabc5d6e38bfc7cdb7b01e117decbdb26ee44a922ba6e"} Sep 16 04:49:25.116655 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:25.116377 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:25.116860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:25.116453 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:25.116860 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:25.116756 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:25.116860 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:25.116489 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:25.117030 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:25.116852 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:25.117030 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:25.116942 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:25.420590 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:25.420519 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="7433880c0cb3e1d1d24e8fa0246ef94440a85b0886a292f523ce7335fc21b93c" exitCode=0 Sep 16 04:49:25.420871 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:25.420606 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"7433880c0cb3e1d1d24e8fa0246ef94440a85b0886a292f523ce7335fc21b93c"} Sep 16 04:49:27.114899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.114865 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:27.115229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.114868 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:27.115229 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.114974 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jhmn" podUID="20993e50-9419-404f-a4b6-0eda4a2afeff" Sep 16 04:49:27.115229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.114997 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:27.115229 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.115083 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gknpb" podUID="70065c80-52d4-4eb5-a606-905ee57917a9" Sep 16 04:49:27.115229 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.115149 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v92kt" podUID="4ed280d1-6f4d-4970-a2b7-bd7ec2be338f" Sep 16 04:49:27.647584 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.647527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:27.647761 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.647618 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:27.647761 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.647655 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:27.647761 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.647693 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:27.647761 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.647720 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret podName:20993e50-9419-404f-a4b6-0eda4a2afeff nodeName:}" failed. No retries permitted until 2026-09-16 04:49:59.647703628 +0000 UTC m=+65.161255158 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret") pod "global-pull-secret-syncer-4jhmn" (UID: "20993e50-9419-404f-a4b6-0eda4a2afeff") : object "kube-system"/"original-pull-secret" not registered Sep 16 04:49:27.647761 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.647741 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs podName:70065c80-52d4-4eb5-a606-905ee57917a9 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:59.647729647 +0000 UTC m=+65.161281170 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs") pod "network-metrics-daemon-gknpb" (UID: "70065c80-52d4-4eb5-a606-905ee57917a9") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 04:49:27.848514 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:27.848483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:27.848774 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.848615 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 04:49:27.848774 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.848629 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 04:49:27.848774 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.848639 2560 projected.go:196] Error preparing data for projected volume kube-api-access-lgs5p for pod openshift-network-diagnostics/network-check-target-v92kt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:27.848774 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:27.848690 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p podName:4ed280d1-6f4d-4970-a2b7-bd7ec2be338f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:59.848675424 +0000 UTC m=+65.362226938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lgs5p" (UniqueName: "kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p") pod "network-check-target-v92kt" (UID: "4ed280d1-6f4d-4970-a2b7-bd7ec2be338f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 04:49:28.765284 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.765262 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-144.ec2.internal" event="NodeReady" Sep 16 04:49:28.765668 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.765392 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 04:49:28.802579 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.802258 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:49:28.851874 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.851842 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql"] Sep 16 04:49:28.853143 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.852582 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.859132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.855085 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 04:49:28.859132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.855357 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 04:49:28.859132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.856747 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 04:49:28.859132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.856832 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tff4l\"" Sep 16 04:49:28.863501 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.863480 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 04:49:28.864377 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.864360 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw"] Sep 16 04:49:28.864607 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.864535 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" Sep 16 04:49:28.867158 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.867138 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hkrfg\"" Sep 16 04:49:28.867264 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.867161 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.867264 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.867235 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.878404 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.878388 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt"] Sep 16 04:49:28.878637 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.878619 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:28.881121 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.881093 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 04:49:28.881215 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.881121 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmcgr\"" Sep 16 04:49:28.881215 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.881102 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.881414 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.881397 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.888602 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.888581 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw"] Sep 16 04:49:28.888695 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.888625 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt"] Sep 16 04:49:28.888695 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.888649 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:49:28.888695 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.888671 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jswl4"] Sep 16 04:49:28.888695 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.888688 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:28.891702 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.891680 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cpg2v\"" Sep 16 04:49:28.891878 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.891861 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 04:49:28.891941 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.891895 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 04:49:28.892037 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.892021 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.892123 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.892109 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.898026 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.898008 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9l49f"] Sep 16 04:49:28.898124 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.898108 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:28.900236 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.900219 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 04:49:28.900326 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.900220 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 04:49:28.900326 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.900313 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbjt\"" Sep 16 04:49:28.911670 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.911650 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k"] Sep 16 04:49:28.911782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.911713 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:28.914211 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.914193 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 04:49:28.914648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.914487 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-v6dnp\"" Sep 16 04:49:28.914648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.914489 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.914648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.914518 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.914648 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.914509 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 04:49:28.920132 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.920115 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 04:49:28.926220 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.926195 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vpvwx"] Sep 16 04:49:28.926376 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.926364 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:28.929435 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.929237 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 04:49:28.929435 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.929345 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bcgjd\"" Sep 16 04:49:28.930456 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.930435 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.930527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.930472 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.930618 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.930584 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 04:49:28.946108 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.945990 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj"] Sep 16 04:49:28.946108 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.946102 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:28.948637 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.948618 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.948729 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.948668 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 04:49:28.948729 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.948687 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 04:49:28.949002 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.948985 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fp56w\"" Sep 16 04:49:28.949762 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.949738 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.954239 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.954220 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 04:49:28.958318 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958422 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958328 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdc2\" (UniqueName: \"kubernetes.io/projected/195e625f-1f88-40b1-848b-8efbaff69d33-kube-api-access-cgdc2\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:28.958422 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958361 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:28.958422 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958414 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m2b\" (UniqueName: \"kubernetes.io/projected/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-kube-api-access-k7m2b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:28.958564 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958530 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958574 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:28.958623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcvzr\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958621 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7tx8\" (UniqueName: \"kubernetes.io/projected/c3953222-bec7-4f4c-80c3-2b51fd9b7ebe-kube-api-access-n7tx8\") pod \"volume-data-source-validator-86695b7cc7-krqql\" (UID: \"c3953222-bec7-4f4c-80c3-2b51fd9b7ebe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" Sep 16 04:49:28.958763 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958641 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958763 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958656 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwg6h\" (UniqueName: \"kubernetes.io/projected/20985f63-e070-448b-8a3a-afd4d9a3b235-kube-api-access-cwg6h\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:28.958763 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958720 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958758 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958786 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e625f-1f88-40b1-848b-8efbaff69d33-tmp-dir\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958824 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958850 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958875 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:28.958910 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958902 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195e625f-1f88-40b1-848b-8efbaff69d33-config-volume\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:28.959115 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.958925 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:28.970146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.970129 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75c67795c4-rktps"] Sep 16 04:49:28.970253 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.970235 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:28.973827 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.973776 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 04:49:28.973913 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.973824 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.973913 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.973878 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 04:49:28.973913 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.973885 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2zfd7\"" Sep 16 04:49:28.974192 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.974173 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.986214 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.986195 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5"] Sep 16 04:49:28.986333 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.986312 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:28.989345 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989322 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 04:49:28.989445 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989414 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 04:49:28.989612 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989593 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-skcqb\"" Sep 16 04:49:28.989726 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989611 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 04:49:28.989726 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989711 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 04:49:28.989951 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989649 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 04:49:28.989951 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:28.989777 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 04:49:29.006331 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.006306 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qkwgl"] Sep 16 04:49:29.006462 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.006447 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.008788 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.008758 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 04:49:29.008873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.008812 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-snrn7\"" Sep 16 04:49:29.008873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.008828 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 04:49:29.028624 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.028602 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5wttx"] Sep 16 04:49:29.028736 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.028722 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.031211 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.031182 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 04:49:29.031318 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.031272 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 04:49:29.031368 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.031344 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cl5jq\"" Sep 16 04:49:29.031427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.031410 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 04:49:29.053118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053090 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jswl4"] Sep 16 04:49:29.053237 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053150 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k"] Sep 16 04:49:29.053237 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053180 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj"] Sep 16 04:49:29.053237 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053203 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql"] Sep 16 04:49:29.053237 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053229 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vpvwx"] Sep 16 04:49:29.053423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053248 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9l49f"] Sep 16 04:49:29.053423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053273 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5wttx"] Sep 16 04:49:29.053423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053292 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75c67795c4-rktps"] Sep 16 04:49:29.053423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053318 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qkwgl"] Sep 16 04:49:29.053423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053338 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5"] Sep 16 04:49:29.053654 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.053458 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" Sep 16 04:49:29.056200 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.056181 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 04:49:29.056434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.056417 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 04:49:29.056527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.056421 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5cwnq\"" Sep 16 04:49:29.059224 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.059298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-stats-auth\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.059298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059249 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpsb\" (UniqueName: \"kubernetes.io/projected/1e08416d-00b8-4621-bd10-c653c8e424e3-kube-api-access-nwpsb\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.059298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059265 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-trusted-ca\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.059298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3f3f78-7e08-4046-8046-b8e0842d1258-serving-cert\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059311 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-serving-cert\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059335 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059356 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e625f-1f88-40b1-848b-8efbaff69d33-tmp-dir\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059418 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75l2\" (UniqueName: \"kubernetes.io/projected/31d4c6fc-9a85-49bd-b99d-bd839996e36d-kube-api-access-f75l2\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059441 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195e625f-1f88-40b1-848b-8efbaff69d33-config-volume\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.059489 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.059502 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:29.059513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwg6h\" (UniqueName: \"kubernetes.io/projected/20985f63-e070-448b-8a3a-afd4d9a3b235-kube-api-access-cwg6h\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059528 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-service-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.059586 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.559566664 +0000 UTC m=+35.073118225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059606 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059652 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lrl\" (UniqueName: \"kubernetes.io/projected/94700653-88d4-48d5-9d5a-4ba8174af04f-kube-api-access-n5lrl\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059688 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31d4c6fc-9a85-49bd-b99d-bd839996e36d-serving-cert\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059836 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195e625f-1f88-40b1-848b-8efbaff69d33-tmp-dir\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.059950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060000 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcvzr\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7tx8\" (UniqueName: \"kubernetes.io/projected/c3953222-bec7-4f4c-80c3-2b51fd9b7ebe-kube-api-access-n7tx8\") pod \"volume-data-source-validator-86695b7cc7-krqql\" (UID: \"c3953222-bec7-4f4c-80c3-2b51fd9b7ebe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060086 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-default-certificate\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da526990-e05b-4d59-b01a-217871f25d23-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.060148 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.060149 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.060209 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.560193054 +0000 UTC m=+35.073744580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060278 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060330 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/195e625f-1f88-40b1-848b-8efbaff69d33-config-volume\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060414 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-config\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060455 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e08416d-00b8-4621-bd10-c653c8e424e3-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3f3f78-7e08-4046-8046-b8e0842d1258-config\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060516 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6nb\" (UniqueName: \"kubernetes.io/projected/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-kube-api-access-4t6nb\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060577 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6r7\" (UniqueName: \"kubernetes.io/projected/bc3f3f78-7e08-4046-8046-b8e0842d1258-kube-api-access-5z6r7\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:29.060873 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060657 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060723 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m2b\" (UniqueName: \"kubernetes.io/projected/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-kube-api-access-k7m2b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-tmp\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060776 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdc2\" (UniqueName: \"kubernetes.io/projected/195e625f-1f88-40b1-848b-8efbaff69d33-kube-api-access-cgdc2\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.060790 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.060857 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.560837503 +0000 UTC m=+35.074389026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060878 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.061433 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.060915 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-snapshots\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.061796 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.061493 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.064452 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.064420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.064528 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.064485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.064528 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.064508 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.080387 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.080364 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdc2\" (UniqueName: \"kubernetes.io/projected/195e625f-1f88-40b1-848b-8efbaff69d33-kube-api-access-cgdc2\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.080815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.080791 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcvzr\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.080953 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.080932 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7tx8\" (UniqueName: \"kubernetes.io/projected/c3953222-bec7-4f4c-80c3-2b51fd9b7ebe-kube-api-access-n7tx8\") pod \"volume-data-source-validator-86695b7cc7-krqql\" (UID: \"c3953222-bec7-4f4c-80c3-2b51fd9b7ebe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" Sep 16 04:49:29.080953 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.080950 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.081162 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.081144 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwg6h\" (UniqueName: \"kubernetes.io/projected/20985f63-e070-448b-8a3a-afd4d9a3b235-kube-api-access-cwg6h\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:29.081350 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.081335 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m2b\" (UniqueName: \"kubernetes.io/projected/b6074ca0-4d90-47b2-8c5e-069b750b9fd9-kube-api-access-k7m2b\") pod \"kube-storage-version-migrator-operator-6c8f84f897-qf8dt\" (UID: \"b6074ca0-4d90-47b2-8c5e-069b750b9fd9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.114759 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.114736 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:29.114848 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.114738 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:29.114925 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.114743 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:29.117483 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.117467 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 04:49:29.117588 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.117468 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gbd6v\"" Sep 16 04:49:29.117588 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.117534 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 04:49:29.117715 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.117594 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpw9m\"" Sep 16 04:49:29.162436 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162269 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-service-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.162582 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.162582 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162480 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lrl\" (UniqueName: \"kubernetes.io/projected/94700653-88d4-48d5-9d5a-4ba8174af04f-kube-api-access-n5lrl\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.162582 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31d4c6fc-9a85-49bd-b99d-bd839996e36d-serving-cert\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.162725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162592 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq2vz\" (UniqueName: \"kubernetes.io/projected/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-kube-api-access-qq2vz\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.162725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-default-certificate\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.162725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162655 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da526990-e05b-4d59-b01a-217871f25d23-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.162725 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.162700 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:29.162725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162710 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59z4d\" (UniqueName: \"kubernetes.io/projected/5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a-kube-api-access-59z4d\") pod \"network-check-source-6f8b768455-5wttx\" (UID: \"5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.162757 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.662738892 +0000 UTC m=+35.176290417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162810 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-config\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162834 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e08416d-00b8-4621-bd10-c653c8e424e3-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3f3f78-7e08-4046-8046-b8e0842d1258-config\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162875 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6nb\" (UniqueName: \"kubernetes.io/projected/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-kube-api-access-4t6nb\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162908 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6r7\" (UniqueName: \"kubernetes.io/projected/bc3f3f78-7e08-4046-8046-b8e0842d1258-kube-api-access-5z6r7\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162931 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162962 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-tmp\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.162996 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.162979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-cert\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163005 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163035 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-snapshots\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163032 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-service-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-stats-auth\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163106 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpsb\" (UniqueName: \"kubernetes.io/projected/1e08416d-00b8-4621-bd10-c653c8e424e3-kube-api-access-nwpsb\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163131 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-trusted-ca\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163159 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3f3f78-7e08-4046-8046-b8e0842d1258-serving-cert\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-serving-cert\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163229 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f75l2\" (UniqueName: \"kubernetes.io/projected/31d4c6fc-9a85-49bd-b99d-bd839996e36d-kube-api-access-f75l2\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163318 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/da526990-e05b-4d59-b01a-217871f25d23-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.163427 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.663410834 +0000 UTC m=+35.176962347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:29.163490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163465 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bc3f3f78-7e08-4046-8046-b8e0842d1258-config\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.164159 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.163580 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:29.164159 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.163623 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.663610314 +0000 UTC m=+35.177161827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:29.164159 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.163733 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-config\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.164159 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.164001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-tmp\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.164159 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.164019 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/31d4c6fc-9a85-49bd-b99d-bd839996e36d-snapshots\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.164425 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.164177 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1e08416d-00b8-4621-bd10-c653c8e424e3-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.164425 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.164266 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:29.164425 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.164325 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:29.66430857 +0000 UTC m=+35.177860101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:29.164425 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.164344 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31d4c6fc-9a85-49bd-b99d-bd839996e36d-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.165381 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.165357 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31d4c6fc-9a85-49bd-b99d-bd839996e36d-serving-cert\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.165768 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.165749 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-trusted-ca\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.166455 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.166435 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-stats-auth\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.166538 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.166437 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-default-certificate\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.166538 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.166462 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc3f3f78-7e08-4046-8046-b8e0842d1258-serving-cert\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.166753 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.166735 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-serving-cert\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.176671 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.176628 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6r7\" (UniqueName: \"kubernetes.io/projected/bc3f3f78-7e08-4046-8046-b8e0842d1258-kube-api-access-5z6r7\") pod \"service-ca-operator-cf7c776b7-t8lzj\" (UID: \"bc3f3f78-7e08-4046-8046-b8e0842d1258\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.176767 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.176691 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lrl\" (UniqueName: \"kubernetes.io/projected/94700653-88d4-48d5-9d5a-4ba8174af04f-kube-api-access-n5lrl\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.177028 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.177010 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6nb\" (UniqueName: \"kubernetes.io/projected/d8f5f528-9fe1-4dcd-a916-f2db73ae6d96-kube-api-access-4t6nb\") pod \"console-operator-696884fb9c-vpvwx\" (UID: \"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96\") " pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.177094 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.177082 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" Sep 16 04:49:29.177281 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.177259 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75l2\" (UniqueName: \"kubernetes.io/projected/31d4c6fc-9a85-49bd-b99d-bd839996e36d-kube-api-access-f75l2\") pod \"insights-operator-5844b856db-9l49f\" (UID: \"31d4c6fc-9a85-49bd-b99d-bd839996e36d\") " pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.177336 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.177270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpsb\" (UniqueName: \"kubernetes.io/projected/1e08416d-00b8-4621-bd10-c653c8e424e3-kube-api-access-nwpsb\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.198075 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.198056 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" Sep 16 04:49:29.221512 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.221492 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9l49f" Sep 16 04:49:29.256183 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.256160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:29.264463 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.264442 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-cert\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.264617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.264599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq2vz\" (UniqueName: \"kubernetes.io/projected/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-kube-api-access-qq2vz\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.264685 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.264650 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59z4d\" (UniqueName: \"kubernetes.io/projected/5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a-kube-api-access-59z4d\") pod \"network-check-source-6f8b768455-5wttx\" (UID: \"5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" Sep 16 04:49:29.267438 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.267415 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-cert\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.272937 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.272912 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59z4d\" (UniqueName: \"kubernetes.io/projected/5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a-kube-api-access-59z4d\") pod \"network-check-source-6f8b768455-5wttx\" (UID: \"5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" Sep 16 04:49:29.273030 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.272966 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq2vz\" (UniqueName: \"kubernetes.io/projected/f13e2f9d-cfab-478b-9ad1-58f6dd8c0336-kube-api-access-qq2vz\") pod \"ingress-canary-qkwgl\" (UID: \"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336\") " pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.280704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.280683 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" Sep 16 04:49:29.336597 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.336569 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qkwgl" Sep 16 04:49:29.363219 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.363198 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" Sep 16 04:49:29.567147 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.567065 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:29.567315 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.567167 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:29.567315 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.567216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:29.567315 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567247 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:29.567315 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567269 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:29.567513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567247 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:29.567513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567333 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.567311443 +0000 UTC m=+36.080862966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:29.567513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567344 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:29.567513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567390 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.567364192 +0000 UTC m=+36.080915708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:29.567513 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.567410 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.567400807 +0000 UTC m=+36.080952320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:29.668273 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.668243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.668451 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.668306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:29.668451 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.668380 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:29.668451 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:29.668410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:29.668451 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668415 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668454 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668485 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.668466872 +0000 UTC m=+36.182018406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668514 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.668493645 +0000 UTC m=+36.182045172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668513 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668539 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.668526262 +0000 UTC m=+36.182077793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:29.668686 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:29.668578 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:30.668567092 +0000 UTC m=+36.182118608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:30.574683 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.574647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.574747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.574811 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574823 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574897 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.574878187 +0000 UTC m=+38.088429700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574923 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574968 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574990 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.574975 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.574959664 +0000 UTC m=+38.088511176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:30.575124 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.575046 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.575033354 +0000 UTC m=+38.088584875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:30.676062 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.676033 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:30.676205 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.676072 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:30.676205 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.676136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:30.676205 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:30.676170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:30.676304 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676207 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.67618651 +0000 UTC m=+38.189738036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:30.676304 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676268 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:30.676375 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676303 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:30.676375 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676326 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.676310689 +0000 UTC m=+38.189862208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:30.676375 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676268 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:30.676375 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676342 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.676333083 +0000 UTC m=+38.189884598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:30.676375 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:30.676358 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:32.676348784 +0000 UTC m=+38.189900296 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:31.022728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.021334 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qkwgl"] Sep 16 04:49:31.025858 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.025270 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9l49f"] Sep 16 04:49:31.027632 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.027118 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql"] Sep 16 04:49:31.029131 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.028862 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj"] Sep 16 04:49:31.029730 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.029666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-5wttx"] Sep 16 04:49:31.031406 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.030990 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt"] Sep 16 04:49:31.033495 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.032058 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-vpvwx"] Sep 16 04:49:31.435394 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.435184 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qkwgl" event={"ID":"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336","Type":"ContainerStarted","Data":"5fcfd5cf3e9ee5f2a1ae4ead609b4b9d95f55f8b867d4a4eb093b2e8d9777d1f"} Sep 16 04:49:31.436226 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.436203 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" event={"ID":"bc3f3f78-7e08-4046-8046-b8e0842d1258","Type":"ContainerStarted","Data":"27a3ee741df665ecd4df387516c0f52c9ba2e4b3ff112ec5eedf75d391204e2f"} Sep 16 04:49:31.437323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.437297 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" event={"ID":"c3953222-bec7-4f4c-80c3-2b51fd9b7ebe","Type":"ContainerStarted","Data":"5a7d72aa328130741074ec87c8aacedc1a7bd8329bfcc3d927291e4bda1a7bf1"} Sep 16 04:49:31.439950 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.439927 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="42c1054e3641bc7bb40c16377f471d569b0d3c2cebae471a762a6b743c5aed72" exitCode=0 Sep 16 04:49:31.440067 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.440005 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"42c1054e3641bc7bb40c16377f471d569b0d3c2cebae471a762a6b743c5aed72"} Sep 16 04:49:31.441216 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.441191 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" event={"ID":"b6074ca0-4d90-47b2-8c5e-069b750b9fd9","Type":"ContainerStarted","Data":"9ac321fd45d230887853580da51696fecccfaa542b5329f68d5ce3c2410235d8"} Sep 16 04:49:31.442297 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.442279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9l49f" event={"ID":"31d4c6fc-9a85-49bd-b99d-bd839996e36d","Type":"ContainerStarted","Data":"6afb4681103c2fd486a9f068e551b5cd153a08341d31c0782784778a930c228e"} Sep 16 04:49:31.443299 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.443268 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" event={"ID":"5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a","Type":"ContainerStarted","Data":"f47b3cdcb8433f3430e5d101201cba936a852dd7dbd83580cf78b177e6ef6b44"} Sep 16 04:49:31.444474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:31.444456 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" event={"ID":"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96","Type":"ContainerStarted","Data":"6b9c6a1f4a813884c3dbf1fcabf3a301849f63985340b59f93088214cdf0d7d6"} Sep 16 04:49:32.461847 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.460982 2560 generic.go:356] "Generic (PLEG): container finished" podID="7439855c-a586-4e18-b668-31e119086964" containerID="3566e240fd25872fcc3d47d5f2f8edd8be0fac8746577d543a87de8203ff4365" exitCode=0 Sep 16 04:49:32.461847 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.461039 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerDied","Data":"3566e240fd25872fcc3d47d5f2f8edd8be0fac8746577d543a87de8203ff4365"} Sep 16 04:49:32.598355 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.598306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:32.598515 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.598432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:32.598515 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.598494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:32.598681 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.598663 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:32.598748 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.598734 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.598715812 +0000 UTC m=+42.112267328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:32.599430 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.599406 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:32.599430 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.599427 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:32.599611 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.599468 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.599455174 +0000 UTC m=+42.113006703 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:32.599878 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.599862 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:32.599960 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.599906 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.599893433 +0000 UTC m=+42.113444945 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:32.699760 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.699723 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:32.699931 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.699840 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:32.699931 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.699876 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:32.700045 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:32.699964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:32.700120 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700103 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:32.700191 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700171 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.700152816 +0000 UTC m=+42.213704345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:32.700608 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700589 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:32.700728 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700643 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.700628415 +0000 UTC m=+42.214179928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:32.700728 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700716 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.700706176 +0000 UTC m=+42.214257691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:32.700850 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700778 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:32.700850 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:32.700811 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:36.700800797 +0000 UTC m=+42.214352309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:33.477313 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:33.476401 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" event={"ID":"7439855c-a586-4e18-b668-31e119086964","Type":"ContainerStarted","Data":"b5d29387c7dcbfdce3a90d47c3e90aa4f2e21539d42d1e719857230f5953fc8b"} Sep 16 04:49:33.501334 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:33.501277 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d4rtn" podStartSLOduration=6.446504802 podStartE2EDuration="37.501254835s" totalImagesPullingTime="31.054750033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:33.499227343 +0000 UTC m=+39.012778902" watchObservedRunningTime="2026-09-16 04:49:33.501254835 +0000 UTC m=+39.014806367" Sep 16 04:49:34.480080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:34.480048 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" event={"ID":"c3953222-bec7-4f4c-80c3-2b51fd9b7ebe","Type":"ContainerStarted","Data":"edc41ebd52f143c2604ea37b9882ab734ac98c6577a4b6abb1fc319af40bda27"} Sep 16 04:49:34.498949 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:34.498899 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-krqql" podStartSLOduration=29.9935828 podStartE2EDuration="32.498878318s" totalImagesPullingTime="2.505295518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:34.497044292 +0000 UTC m=+40.010595837" watchObservedRunningTime="2026-09-16 04:49:34.498878318 +0000 UTC m=+40.012429854" Sep 16 04:49:36.639152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.639116 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.639174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.639262 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639291 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639357 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.639338702 +0000 UTC m=+50.152890217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639373 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639399 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639410 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639443 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.639425287 +0000 UTC m=+50.152976816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:36.639602 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.639463 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.639451971 +0000 UTC m=+50.153003491 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:36.740420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.740385 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:36.740605 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.740435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:36.740605 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.740526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:36.740722 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740602 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.740579757 +0000 UTC m=+50.254131286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:36.740722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:36.740639 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:36.740722 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740666 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:36.740722 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740718 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.740703876 +0000 UTC m=+50.254255414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:36.740918 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740776 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:36.740918 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740804 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.740795705 +0000 UTC m=+50.254347217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:36.740918 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740777 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:36.740918 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:36.740831 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.740825578 +0000 UTC m=+50.254377090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:39.501146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.501107 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" event={"ID":"b6074ca0-4d90-47b2-8c5e-069b750b9fd9","Type":"ContainerStarted","Data":"1c5f4c20f4451889b0878768f418a9eb70732c06229805e6b00614e931560e20"} Sep 16 04:49:39.503488 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.503197 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9l49f" event={"ID":"31d4c6fc-9a85-49bd-b99d-bd839996e36d","Type":"ContainerStarted","Data":"c5df1d5d86ede83b76bbea889177dd201312bf494a0f0c387e2f32111f1bfda3"} Sep 16 04:49:39.505178 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.504850 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" event={"ID":"5b3707a7-07b1-45d8-b4aa-1f0cab0fa09a","Type":"ContainerStarted","Data":"7be78cb953c55a8734936ca1437b7bb33907c6b640717916ac3b0cc9a5dab258"} Sep 16 04:49:39.507142 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.507121 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/0.log" Sep 16 04:49:39.507264 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.507159 2560 generic.go:356] "Generic (PLEG): container finished" podID="d8f5f528-9fe1-4dcd-a916-f2db73ae6d96" containerID="760936f3731777e34c156fa23f6dbf86855b65afe7620b4fd31d9e02d546c231" exitCode=255 Sep 16 04:49:39.507264 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.507185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" event={"ID":"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96","Type":"ContainerDied","Data":"760936f3731777e34c156fa23f6dbf86855b65afe7620b4fd31d9e02d546c231"} Sep 16 04:49:39.507405 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.507386 2560 scope.go:117] "RemoveContainer" containerID="760936f3731777e34c156fa23f6dbf86855b65afe7620b4fd31d9e02d546c231" Sep 16 04:49:39.509384 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.509360 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qkwgl" event={"ID":"f13e2f9d-cfab-478b-9ad1-58f6dd8c0336","Type":"ContainerStarted","Data":"4558e4d57b31780fc0acdb9bfc213ac6102bd57c3e128328c00277656819fd04"} Sep 16 04:49:39.510877 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.510855 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" event={"ID":"bc3f3f78-7e08-4046-8046-b8e0842d1258","Type":"ContainerStarted","Data":"4187c06f7ba27789e578166fdf0e3b1c9e3be43dbf286a1be68daf3cc519e6e8"} Sep 16 04:49:39.524306 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.524271 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" podStartSLOduration=29.717710103 podStartE2EDuration="37.524245524s" totalImagesPullingTime="7.806535421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:39.523634004 +0000 UTC m=+45.037185539" watchObservedRunningTime="2026-09-16 04:49:39.524245524 +0000 UTC m=+45.037797059" Sep 16 04:49:39.562228 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.560921 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9l49f" podStartSLOduration=29.754950443 podStartE2EDuration="37.560902719s" totalImagesPullingTime="7.805952276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:39.542803054 +0000 UTC m=+45.056354591" watchObservedRunningTime="2026-09-16 04:49:39.560902719 +0000 UTC m=+45.074454251" Sep 16 04:49:39.576226 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.576179 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" podStartSLOduration=29.772190942 podStartE2EDuration="37.576147733s" totalImagesPullingTime="7.803956791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:39.575089866 +0000 UTC m=+45.088641401" watchObservedRunningTime="2026-09-16 04:49:39.576147733 +0000 UTC m=+45.089699268" Sep 16 04:49:39.590566 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.589922 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-5wttx" podStartSLOduration=29.788400166 podStartE2EDuration="37.589909485s" totalImagesPullingTime="7.801509319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:39.588816422 +0000 UTC m=+45.102367957" watchObservedRunningTime="2026-09-16 04:49:39.589909485 +0000 UTC m=+45.103461018" Sep 16 04:49:39.606687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:39.606649 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qkwgl" podStartSLOduration=3.801006282 podStartE2EDuration="11.606631053s" totalImagesPullingTime="7.805624771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:39.604087182 +0000 UTC m=+45.117638717" watchObservedRunningTime="2026-09-16 04:49:39.606631053 +0000 UTC m=+45.120182589" Sep 16 04:49:40.297223 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.297128 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p7fmq"] Sep 16 04:49:40.319620 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.319597 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7fmq"] Sep 16 04:49:40.319782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.319725 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.322239 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.322213 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 04:49:40.322505 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.322489 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 04:49:40.322579 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.322495 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jkwpt\"" Sep 16 04:49:40.474369 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.474338 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a86b362-5097-4439-8d31-69ef7ccce013-crio-socket\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.474591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.474404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.474591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.474526 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.474591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.474570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a86b362-5097-4439-8d31-69ef7ccce013-data-volume\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.474734 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.474606 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxv9\" (UniqueName: \"kubernetes.io/projected/6a86b362-5097-4439-8d31-69ef7ccce013-kube-api-access-5cxv9\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.515364 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.515340 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:49:40.515821 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.515807 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/0.log" Sep 16 04:49:40.515872 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.515841 2560 generic.go:356] "Generic (PLEG): container finished" podID="d8f5f528-9fe1-4dcd-a916-f2db73ae6d96" containerID="64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638" exitCode=255 Sep 16 04:49:40.515926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.515911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" event={"ID":"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96","Type":"ContainerDied","Data":"64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638"} Sep 16 04:49:40.515976 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.515958 2560 scope.go:117] "RemoveContainer" containerID="760936f3731777e34c156fa23f6dbf86855b65afe7620b4fd31d9e02d546c231" Sep 16 04:49:40.516155 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.516130 2560 scope.go:117] "RemoveContainer" containerID="64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638" Sep 16 04:49:40.516341 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:40.516316 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vpvwx_openshift-console-operator(d8f5f528-9fe1-4dcd-a916-f2db73ae6d96)\"" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" podUID="d8f5f528-9fe1-4dcd-a916-f2db73ae6d96" Sep 16 04:49:40.575886 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.575862 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a86b362-5097-4439-8d31-69ef7ccce013-data-volume\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576006 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.575899 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxv9\" (UniqueName: \"kubernetes.io/projected/6a86b362-5097-4439-8d31-69ef7ccce013-kube-api-access-5cxv9\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576078 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.576055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a86b362-5097-4439-8d31-69ef7ccce013-crio-socket\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576136 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.576117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576223 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.576203 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576223 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.576210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6a86b362-5097-4439-8d31-69ef7ccce013-data-volume\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576348 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.576228 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6a86b362-5097-4439-8d31-69ef7ccce013-crio-socket\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:40.576348 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:40.576266 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 04:49:40.576348 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:40.576316 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:41.076304143 +0000 UTC m=+46.589855655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : configmap "kube-rbac-proxy" not found Sep 16 04:49:40.576497 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:40.576267 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 04:49:40.576497 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:40.576409 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:41.076393134 +0000 UTC m=+46.589944653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : secret "insights-runtime-extractor-tls" not found Sep 16 04:49:40.588047 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:40.588023 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxv9\" (UniqueName: \"kubernetes.io/projected/6a86b362-5097-4439-8d31-69ef7ccce013-kube-api-access-5cxv9\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:41.081301 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:41.081270 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:41.081438 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:41.081411 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 04:49:41.081438 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:41.081414 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:41.081582 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:41.081474 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:42.081456103 +0000 UTC m=+47.595007638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : configmap "kube-rbac-proxy" not found Sep 16 04:49:41.081582 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:41.081521 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 04:49:41.081582 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:41.081579 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:42.081566367 +0000 UTC m=+47.595117896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : secret "insights-runtime-extractor-tls" not found Sep 16 04:49:41.521159 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:41.521084 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:49:41.521509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:41.521386 2560 scope.go:117] "RemoveContainer" containerID="64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638" Sep 16 04:49:41.521581 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:41.521562 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vpvwx_openshift-console-operator(d8f5f528-9fe1-4dcd-a916-f2db73ae6d96)\"" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" podUID="d8f5f528-9fe1-4dcd-a916-f2db73ae6d96" Sep 16 04:49:42.038986 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:42.038959 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fx8ph_695af7a0-6dde-4d7d-b185-32ef2b342741/dns-node-resolver/0.log" Sep 16 04:49:42.090601 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:42.090571 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:42.090748 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:42.090711 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 04:49:42.090811 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:42.090755 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:42.090811 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:42.090765 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.090751452 +0000 UTC m=+49.604302964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : secret "insights-runtime-extractor-tls" not found Sep 16 04:49:42.090929 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:42.090861 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 04:49:42.090973 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:42.090930 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:44.090912062 +0000 UTC m=+49.604463589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : configmap "kube-rbac-proxy" not found Sep 16 04:49:43.038878 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:43.038852 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vl57t_6155ff74-4c91-44f2-8bd7-6768c8fe3de8/node-ca/0.log" Sep 16 04:49:43.639150 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:43.639122 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qkwgl_f13e2f9d-cfab-478b-9ad1-58f6dd8c0336/serve-healthcheck-canary/0.log" Sep 16 04:49:44.110279 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.110249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:44.110662 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.110355 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:44.110662 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.110374 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 04:49:44.110662 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.110434 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:48.110421065 +0000 UTC m=+53.623972581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : configmap "kube-rbac-proxy" not found Sep 16 04:49:44.110662 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.110449 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 04:49:44.110662 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.110488 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:48.110476957 +0000 UTC m=+53.624028469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : secret "insights-runtime-extractor-tls" not found Sep 16 04:49:44.714641 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.714610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:49:44.714817 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.714678 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:49:44.714817 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.714719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:49:44.714817 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714765 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 04:49:44.714817 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714802 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 04:49:44.714962 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714826 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls podName:195e625f-1f88-40b1-848b-8efbaff69d33 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.714811118 +0000 UTC m=+66.228362654 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls") pod "dns-default-jswl4" (UID: "195e625f-1f88-40b1-848b-8efbaff69d33") : secret "dns-default-metrics-tls" not found Sep 16 04:49:44.714962 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714772 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 04:49:44.714962 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714840 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f67679f5d-pk5v2: secret "image-registry-tls" not found Sep 16 04:49:44.714962 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714840 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls podName:20985f63-e070-448b-8a3a-afd4d9a3b235 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.714833668 +0000 UTC m=+66.228385179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-z7wvw" (UID: "20985f63-e070-448b-8a3a-afd4d9a3b235") : secret "samples-operator-tls" not found Sep 16 04:49:44.714962 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.714870 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls podName:93a70d1f-afc4-40d6-b8de-bbc5f98de9ca nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.714860644 +0000 UTC m=+66.228412156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls") pod "image-registry-7f67679f5d-pk5v2" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca") : secret "image-registry-tls" not found Sep 16 04:49:44.815536 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.815503 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:44.815536 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.815537 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.815612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815628 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.815610933 +0000 UTC m=+66.329162464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : configmap references non-existent config key: service-ca.crt Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815677 2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815705 2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815722 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs podName:94700653-88d4-48d5-9d5a-4ba8174af04f nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.815707671 +0000 UTC m=+66.329259202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs") pod "router-default-75c67795c4-rktps" (UID: "94700653-88d4-48d5-9d5a-4ba8174af04f") : secret "router-metrics-certs-default" not found Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:44.815704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815743 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert podName:da526990-e05b-4d59-b01a-217871f25d23 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.815733414 +0000 UTC m=+66.329284931 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-bg5v5" (UID: "da526990-e05b-4d59-b01a-217871f25d23") : secret "networking-console-plugin-cert" not found Sep 16 04:49:44.815777 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815773 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:44.816056 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:44.815813 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls podName:1e08416d-00b8-4621-bd10-c653c8e424e3 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:00.815801173 +0000 UTC m=+66.329352702 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-sfq6k" (UID: "1e08416d-00b8-4621-bd10-c653c8e424e3") : secret "cluster-monitoring-operator-tls" not found Sep 16 04:49:48.144836 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:48.144802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:48.145223 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:48.144940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:48.145223 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:48.145073 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 04:49:48.145223 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:48.145134 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls podName:6a86b362-5097-4439-8d31-69ef7ccce013 nodeName:}" failed. No retries permitted until 2026-09-16 04:49:56.14512004 +0000 UTC m=+61.658671586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p7fmq" (UID: "6a86b362-5097-4439-8d31-69ef7ccce013") : secret "insights-runtime-extractor-tls" not found Sep 16 04:49:48.145394 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:48.145376 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6a86b362-5097-4439-8d31-69ef7ccce013-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:49.257208 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:49.257173 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:49.257663 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:49.257234 2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:49:49.257663 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:49.257543 2560 scope.go:117] "RemoveContainer" containerID="64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638" Sep 16 04:49:49.257735 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:49:49.257709 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-vpvwx_openshift-console-operator(d8f5f528-9fe1-4dcd-a916-f2db73ae6d96)\"" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" podUID="d8f5f528-9fe1-4dcd-a916-f2db73ae6d96" Sep 16 04:49:55.431473 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:55.431447 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r2bcd" Sep 16 04:49:56.210030 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.209972 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:56.212347 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.212324 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6a86b362-5097-4439-8d31-69ef7ccce013-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7fmq\" (UID: \"6a86b362-5097-4439-8d31-69ef7ccce013\") " pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:56.232328 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.232303 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jkwpt\"" Sep 16 04:49:56.240527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.240505 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7fmq" Sep 16 04:49:56.510240 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.510123 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7fmq"] Sep 16 04:49:56.512699 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:49:56.512667 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a86b362_5097_4439_8d31_69ef7ccce013.slice/crio-c8854ae8ec393e68bc56cfa074996ec69e0465c35469c3d455e649fc41d1bfef WatchSource:0}: Error finding container c8854ae8ec393e68bc56cfa074996ec69e0465c35469c3d455e649fc41d1bfef: Status 404 returned error can't find the container with id c8854ae8ec393e68bc56cfa074996ec69e0465c35469c3d455e649fc41d1bfef Sep 16 04:49:56.564508 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:56.564419 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7fmq" event={"ID":"6a86b362-5097-4439-8d31-69ef7ccce013","Type":"ContainerStarted","Data":"c8854ae8ec393e68bc56cfa074996ec69e0465c35469c3d455e649fc41d1bfef"} Sep 16 04:49:57.569298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:57.569204 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7fmq" event={"ID":"6a86b362-5097-4439-8d31-69ef7ccce013","Type":"ContainerStarted","Data":"233480f1324a18d1c8ed84e86651a5c0a7cd716989988a06d44b561669b3e06d"} Sep 16 04:49:57.569298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:57.569268 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7fmq" event={"ID":"6a86b362-5097-4439-8d31-69ef7ccce013","Type":"ContainerStarted","Data":"25c0fc0574ea82192e1c1f15edf27943ee623cdf3f09bccf1a4e5950ce1831ef"} Sep 16 04:49:59.577031 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.576994 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7fmq" event={"ID":"6a86b362-5097-4439-8d31-69ef7ccce013","Type":"ContainerStarted","Data":"d69b73c00d3939b598f868d14b4fff3148ca85aab5f82c0d1c969b3534f44576"} Sep 16 04:49:59.594670 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.594627 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p7fmq" podStartSLOduration=17.594010196 podStartE2EDuration="19.594611114s" totalImagesPullingTime="2.000600918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:49:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:49:59.593269846 +0000 UTC m=+65.106821379" watchObservedRunningTime="2026-09-16 04:49:59.594611114 +0000 UTC m=+65.108162647" Sep 16 04:49:59.742651 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.742610 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:59.742822 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.742666 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:59.745388 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.745365 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 04:49:59.745388 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.745379 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 04:49:59.755769 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.755741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/20993e50-9419-404f-a4b6-0eda4a2afeff-original-pull-secret\") pod \"global-pull-secret-syncer-4jhmn\" (UID: \"20993e50-9419-404f-a4b6-0eda4a2afeff\") " pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:49:59.755769 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.755762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70065c80-52d4-4eb5-a606-905ee57917a9-metrics-certs\") pod \"network-metrics-daemon-gknpb\" (UID: \"70065c80-52d4-4eb5-a606-905ee57917a9\") " pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:49:59.945142 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.945054 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:49:59.947110 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:49:59.947089 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgs5p\" (UniqueName: \"kubernetes.io/projected/4ed280d1-6f4d-4970-a2b7-bd7ec2be338f-kube-api-access-lgs5p\") pod \"network-check-target-v92kt\" (UID: \"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f\") " pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:50:00.023973 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.023934 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jhmn" Sep 16 04:50:00.032765 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.032744 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mpw9m\"" Sep 16 04:50:00.041538 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.037476 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gbd6v\"" Sep 16 04:50:00.041538 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.041309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gknpb" Sep 16 04:50:00.044927 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.044903 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:50:00.299932 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.299860 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jhmn"] Sep 16 04:50:00.302212 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:00.302185 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20993e50_9419_404f_a4b6_0eda4a2afeff.slice/crio-91a09194b8cedce2db682f0c2be8b1bc37379c7c6d9af50684fb51ea2a47b2bc WatchSource:0}: Error finding container 91a09194b8cedce2db682f0c2be8b1bc37379c7c6d9af50684fb51ea2a47b2bc: Status 404 returned error can't find the container with id 91a09194b8cedce2db682f0c2be8b1bc37379c7c6d9af50684fb51ea2a47b2bc Sep 16 04:50:00.327453 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.327429 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v92kt"] Sep 16 04:50:00.330002 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:00.329971 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed280d1_6f4d_4970_a2b7_bd7ec2be338f.slice/crio-13c3b1a7edbcc70a818ab91340092bb1d762803f42bc70a65c1f37cb96d873ae WatchSource:0}: Error finding container 13c3b1a7edbcc70a818ab91340092bb1d762803f42bc70a65c1f37cb96d873ae: Status 404 returned error can't find the container with id 13c3b1a7edbcc70a818ab91340092bb1d762803f42bc70a65c1f37cb96d873ae Sep 16 04:50:00.330647 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.330628 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gknpb"] Sep 16 04:50:00.332672 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:00.332653 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70065c80_52d4_4eb5_a606_905ee57917a9.slice/crio-2f8d1e73c5f5d83a3d3f50f06004d69ead274c3a1d059c9fb32cb54d1bba90f9 WatchSource:0}: Error finding container 2f8d1e73c5f5d83a3d3f50f06004d69ead274c3a1d059c9fb32cb54d1bba90f9: Status 404 returned error can't find the container with id 2f8d1e73c5f5d83a3d3f50f06004d69ead274c3a1d059c9fb32cb54d1bba90f9 Sep 16 04:50:00.583990 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.583954 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jhmn" event={"ID":"20993e50-9419-404f-a4b6-0eda4a2afeff","Type":"ContainerStarted","Data":"91a09194b8cedce2db682f0c2be8b1bc37379c7c6d9af50684fb51ea2a47b2bc"} Sep 16 04:50:00.585046 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.585023 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gknpb" event={"ID":"70065c80-52d4-4eb5-a606-905ee57917a9","Type":"ContainerStarted","Data":"2f8d1e73c5f5d83a3d3f50f06004d69ead274c3a1d059c9fb32cb54d1bba90f9"} Sep 16 04:50:00.586339 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.586316 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v92kt" event={"ID":"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f","Type":"ContainerStarted","Data":"90a0e617591c40b63572f0a55d8a54bff8831d30c6dc33f6a80a2c31fb2cfa46"} Sep 16 04:50:00.586379 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.586362 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v92kt" event={"ID":"4ed280d1-6f4d-4970-a2b7-bd7ec2be338f","Type":"ContainerStarted","Data":"13c3b1a7edbcc70a818ab91340092bb1d762803f42bc70a65c1f37cb96d873ae"} Sep 16 04:50:00.602042 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.602000 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v92kt" podStartSLOduration=64.60198692 podStartE2EDuration="1m4.60198692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:00.601035297 +0000 UTC m=+66.114586829" watchObservedRunningTime="2026-09-16 04:50:00.60198692 +0000 UTC m=+66.115538453" Sep 16 04:50:00.751464 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.751424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:50:00.751654 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.751508 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:00.751654 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.751576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:50:00.753869 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.753837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"image-registry-7f67679f5d-pk5v2\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:00.753869 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.753850 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/195e625f-1f88-40b1-848b-8efbaff69d33-metrics-tls\") pod \"dns-default-jswl4\" (UID: \"195e625f-1f88-40b1-848b-8efbaff69d33\") " pod="openshift-dns/dns-default-jswl4" Sep 16 04:50:00.754012 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.753925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/20985f63-e070-448b-8a3a-afd4d9a3b235-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-z7wvw\" (UID: \"20985f63-e070-448b-8a3a-afd4d9a3b235\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:50:00.852181 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.852091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:00.852181 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.852139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:50:00.852393 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.852201 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:00.852393 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.852238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:50:00.852723 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.852693 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94700653-88d4-48d5-9d5a-4ba8174af04f-service-ca-bundle\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:00.854560 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.854509 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/da526990-e05b-4d59-b01a-217871f25d23-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-bg5v5\" (UID: \"da526990-e05b-4d59-b01a-217871f25d23\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:50:00.854560 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.854536 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94700653-88d4-48d5-9d5a-4ba8174af04f-metrics-certs\") pod \"router-default-75c67795c4-rktps\" (UID: \"94700653-88d4-48d5-9d5a-4ba8174af04f\") " pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:00.854703 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.854513 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e08416d-00b8-4621-bd10-c653c8e424e3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-sfq6k\" (UID: \"1e08416d-00b8-4621-bd10-c653c8e424e3\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:50:00.974441 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.974412 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tff4l\"" Sep 16 04:50:00.982479 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.982453 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:00.990906 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.990884 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mmcgr\"" Sep 16 04:50:00.999084 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:00.998897 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" Sep 16 04:50:01.008675 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.008479 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbjt\"" Sep 16 04:50:01.016802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.016612 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jswl4" Sep 16 04:50:01.041977 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.041937 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bcgjd\"" Sep 16 04:50:01.049336 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.049309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" Sep 16 04:50:01.102613 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.102539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-skcqb\"" Sep 16 04:50:01.108000 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.107828 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:01.118654 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.118631 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-snrn7\"" Sep 16 04:50:01.127014 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.126480 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" Sep 16 04:50:01.270637 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.270606 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:50:01.273414 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:01.273384 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a70d1f_afc4_40d6_b8de_bbc5f98de9ca.slice/crio-8435e34dbdab3822a8b3837b1838a8385e3e5cab8394c0489fa19117b02953b1 WatchSource:0}: Error finding container 8435e34dbdab3822a8b3837b1838a8385e3e5cab8394c0489fa19117b02953b1: Status 404 returned error can't find the container with id 8435e34dbdab3822a8b3837b1838a8385e3e5cab8394c0489fa19117b02953b1 Sep 16 04:50:01.292728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.292702 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw"] Sep 16 04:50:01.317284 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.317209 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jswl4"] Sep 16 04:50:01.354106 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.354035 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k"] Sep 16 04:50:01.417820 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.417790 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75c67795c4-rktps"] Sep 16 04:50:01.421080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.421027 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5"] Sep 16 04:50:01.528005 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:01.527962 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195e625f_1f88_40b1_848b_8efbaff69d33.slice/crio-c34f51198120c22c0a97c8e872d22cf12093afe181e87e1c168b8c57833c3351 WatchSource:0}: Error finding container c34f51198120c22c0a97c8e872d22cf12093afe181e87e1c168b8c57833c3351: Status 404 returned error can't find the container with id c34f51198120c22c0a97c8e872d22cf12093afe181e87e1c168b8c57833c3351 Sep 16 04:50:01.532079 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:01.529185 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda526990_e05b_4d59_b01a_217871f25d23.slice/crio-b8dc057ff13711b4dcd719dc2d38776ffc396312abfbeeb008f5f9915e172cae WatchSource:0}: Error finding container b8dc057ff13711b4dcd719dc2d38776ffc396312abfbeeb008f5f9915e172cae: Status 404 returned error can't find the container with id b8dc057ff13711b4dcd719dc2d38776ffc396312abfbeeb008f5f9915e172cae Sep 16 04:50:01.533777 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:01.533743 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e08416d_00b8_4621_bd10_c653c8e424e3.slice/crio-9d9bea82db2696b9e67407b4677bcd869de03e52f065b13eb3e1821c88916c5f WatchSource:0}: Error finding container 9d9bea82db2696b9e67407b4677bcd869de03e52f065b13eb3e1821c88916c5f: Status 404 returned error can't find the container with id 9d9bea82db2696b9e67407b4677bcd869de03e52f065b13eb3e1821c88916c5f Sep 16 04:50:01.593127 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.593097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" event={"ID":"1e08416d-00b8-4621-bd10-c653c8e424e3","Type":"ContainerStarted","Data":"9d9bea82db2696b9e67407b4677bcd869de03e52f065b13eb3e1821c88916c5f"} Sep 16 04:50:01.594652 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.594604 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" event={"ID":"20985f63-e070-448b-8a3a-afd4d9a3b235","Type":"ContainerStarted","Data":"3291ff20c4c7357515ec521bb15b7f14d2947ae6fe95c22662c94a88350ba2f1"} Sep 16 04:50:01.596887 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.596760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" event={"ID":"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca","Type":"ContainerStarted","Data":"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04"} Sep 16 04:50:01.596887 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.596802 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" event={"ID":"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca","Type":"ContainerStarted","Data":"8435e34dbdab3822a8b3837b1838a8385e3e5cab8394c0489fa19117b02953b1"} Sep 16 04:50:01.599385 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.599314 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" event={"ID":"da526990-e05b-4d59-b01a-217871f25d23","Type":"ContainerStarted","Data":"b8dc057ff13711b4dcd719dc2d38776ffc396312abfbeeb008f5f9915e172cae"} Sep 16 04:50:01.601502 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.601481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75c67795c4-rktps" event={"ID":"94700653-88d4-48d5-9d5a-4ba8174af04f","Type":"ContainerStarted","Data":"55b2fbc67fdf9c03a9317876f0ff17b05e05c7c1ccec1a6d21ba1ead381a369a"} Sep 16 04:50:01.603725 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.603694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jswl4" event={"ID":"195e625f-1f88-40b1-848b-8efbaff69d33","Type":"ContainerStarted","Data":"c34f51198120c22c0a97c8e872d22cf12093afe181e87e1c168b8c57833c3351"} Sep 16 04:50:01.604007 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.603936 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:50:01.622899 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:01.622851 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75c67795c4-rktps" podStartSLOduration=59.622831306 podStartE2EDuration="59.622831306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:01.620261981 +0000 UTC m=+67.133813516" watchObservedRunningTime="2026-09-16 04:50:01.622831306 +0000 UTC m=+67.136382845" Sep 16 04:50:02.109227 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.109198 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:02.111965 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.111940 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:02.619289 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.619248 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75c67795c4-rktps" event={"ID":"94700653-88d4-48d5-9d5a-4ba8174af04f","Type":"ContainerStarted","Data":"c3090f1dba4a6566fd71f0793ed301521cc0016237f9eb38950cea4a87c8a714"} Sep 16 04:50:02.620333 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.620243 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:02.621883 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.621860 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75c67795c4-rktps" Sep 16 04:50:02.626610 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.626580 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gknpb" event={"ID":"70065c80-52d4-4eb5-a606-905ee57917a9","Type":"ContainerStarted","Data":"0caa504ef7b246b620714b07931798bb7511b665b1d02fcd14048bc115a75945"} Sep 16 04:50:02.626733 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.626635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gknpb" event={"ID":"70065c80-52d4-4eb5-a606-905ee57917a9","Type":"ContainerStarted","Data":"c63ea14e44dde8e7b3c99dec1af621c306670ea0e12dba9f19745269cce3f430"} Sep 16 04:50:02.627113 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.627094 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:02.661453 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.660725 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gknpb" podStartSLOduration=66.408932914 podStartE2EDuration="1m7.660704756s" totalImagesPullingTime="1.251771842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:02.65951893 +0000 UTC m=+68.173070464" watchObservedRunningTime="2026-09-16 04:50:02.660704756 +0000 UTC m=+68.174256290" Sep 16 04:50:02.680121 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:02.679791 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" podStartSLOduration=66.679768684 podStartE2EDuration="1m6.679768684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:02.677782216 +0000 UTC m=+68.191333751" watchObservedRunningTime="2026-09-16 04:50:02.679768684 +0000 UTC m=+68.193320221" Sep 16 04:50:03.124699 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:03.124667 2560 scope.go:117] "RemoveContainer" containerID="64099931b0a09a2c0925a581af7aa73c3fc70dcb92f15f1fd567a9955de7e638" Sep 16 04:50:07.646564 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.646522 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jhmn" event={"ID":"20993e50-9419-404f-a4b6-0eda4a2afeff","Type":"ContainerStarted","Data":"88b1ce82816a6c5aed3af1a676c08200c23f56a2a9b6de0c22ff25eb6cd93722"} Sep 16 04:50:07.648013 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.647988 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" event={"ID":"da526990-e05b-4d59-b01a-217871f25d23","Type":"ContainerStarted","Data":"320c4c6cc7ae82d9234be59f9af5cfe066c4d0f3adcd5a9006a10e7aa245a5e8"} Sep 16 04:50:07.649719 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.649696 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:50:07.649802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.649792 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" event={"ID":"d8f5f528-9fe1-4dcd-a916-f2db73ae6d96","Type":"ContainerStarted","Data":"60685bc34dc9fd4d0c53c590598ba29f47b8cfbd34c958d60314b822a8d7b409"} Sep 16 04:50:07.650095 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.650070 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:50:07.651625 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.651592 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jswl4" event={"ID":"195e625f-1f88-40b1-848b-8efbaff69d33","Type":"ContainerStarted","Data":"c2bb0c71db4f85b96c2133e43e9f676baaa1455c65b67222bb383750f17df329"} Sep 16 04:50:07.651721 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.651651 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jswl4" event={"ID":"195e625f-1f88-40b1-848b-8efbaff69d33","Type":"ContainerStarted","Data":"97fb51a79fc391295be9fa16c67b76727c7c4001da2e2f4ceb0c9d85ab563ee3"} Sep 16 04:50:07.651721 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.651703 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jswl4" Sep 16 04:50:07.653158 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.653136 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" event={"ID":"1e08416d-00b8-4621-bd10-c653c8e424e3","Type":"ContainerStarted","Data":"d6510d66cf67d74acc7c1dfcfb4b4b67b9748a264fb7df4d5c4a889209f5a3f4"} Sep 16 04:50:07.655339 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.655318 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" Sep 16 04:50:07.655420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.655367 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" event={"ID":"20985f63-e070-448b-8a3a-afd4d9a3b235","Type":"ContainerStarted","Data":"f53c534fc2a62b9de92a36187050add99c21d8a633a9e470e835a093a6c0576d"} Sep 16 04:50:07.655420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.655399 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" event={"ID":"20985f63-e070-448b-8a3a-afd4d9a3b235","Type":"ContainerStarted","Data":"e2d3c77126fdfa605b43fa4e48db71a5a55cc96b8f19d411f7f4888380eae0f7"} Sep 16 04:50:07.662296 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.662258 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4jhmn" podStartSLOduration=65.24140211 podStartE2EDuration="1m11.662246585s" totalImagesPullingTime="6.420844475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 04:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.661226814 +0000 UTC m=+73.174778351" watchObservedRunningTime="2026-09-16 04:50:07.662246585 +0000 UTC m=+73.175798125" Sep 16 04:50:07.686765 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.686728 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jswl4" podStartSLOduration=34.498960036 podStartE2EDuration="39.686715402s" totalImagesPullingTime="5.187755366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.685803827 +0000 UTC m=+73.199355373" watchObservedRunningTime="2026-09-16 04:50:07.686715402 +0000 UTC m=+73.200266931" Sep 16 04:50:07.702082 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.702037 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-bg5v5" podStartSLOduration=40.514091909 podStartE2EDuration="45.702021632s" totalImagesPullingTime="5.187929723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.700739429 +0000 UTC m=+73.214290974" watchObservedRunningTime="2026-09-16 04:50:07.702021632 +0000 UTC m=+73.215573166" Sep 16 04:50:07.719213 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.719176 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-z7wvw" podStartSLOduration=60.571493383 podStartE2EDuration="1m5.719163206s" totalImagesPullingTime="5.147669823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.718313532 +0000 UTC m=+73.231865077" watchObservedRunningTime="2026-09-16 04:50:07.719163206 +0000 UTC m=+73.232714739" Sep 16 04:50:07.737952 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.737909 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-sfq6k" podStartSLOduration=60.550339117 podStartE2EDuration="1m5.737892918s" totalImagesPullingTime="5.187553801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.73658604 +0000 UTC m=+73.250137598" watchObservedRunningTime="2026-09-16 04:50:07.737892918 +0000 UTC m=+73.251444452" Sep 16 04:50:07.763096 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:07.763049 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-vpvwx" podStartSLOduration=57.961206499 podStartE2EDuration="1m5.76303315s" totalImagesPullingTime="7.801826651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:07.762486163 +0000 UTC m=+73.276037725" watchObservedRunningTime="2026-09-16 04:50:07.76303315 +0000 UTC m=+73.276584684" Sep 16 04:50:17.663240 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:17.663140 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jswl4" Sep 16 04:50:21.793977 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.793945 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz"] Sep 16 04:50:21.798432 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.798413 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:21.801142 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.801118 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 04:50:21.801248 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.801143 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cl8kt\"" Sep 16 04:50:21.802128 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.802110 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 04:50:21.802227 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.802117 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 04:50:21.811117 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.811097 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz"] Sep 16 04:50:21.812440 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.812422 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf"] Sep 16 04:50:21.815644 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.815626 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.816403 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.816379 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tqf6r"] Sep 16 04:50:21.819430 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.819410 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.821078 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.821061 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zzzgp\"" Sep 16 04:50:21.821299 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.821284 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 04:50:21.821502 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.821483 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 04:50:21.821617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.821607 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 04:50:21.821976 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.821955 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 04:50:21.822587 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.822568 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k8987\"" Sep 16 04:50:21.822696 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.822579 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 04:50:21.822756 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.822648 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 04:50:21.833974 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.833952 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf"] Sep 16 04:50:21.913598 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913570 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.913598 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913600 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnbj\" (UniqueName: \"kubernetes.io/projected/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-kube-api-access-tqnbj\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913617 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-sys\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-root\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913651 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913670 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-textfile\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913718 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-metrics-client-ca\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.913786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0979b33-ec33-4fb0-8e9f-043991d0e998-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2tj\" (UniqueName: \"kubernetes.io/projected/a0979b33-ec33-4fb0-8e9f-043991d0e998-kube-api-access-2m2tj\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913893 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913923 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913939 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdp7g\" (UniqueName: \"kubernetes.io/projected/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-api-access-fdp7g\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-wtmp\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:21.914055 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:21.913987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015258 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015230 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnbj\" (UniqueName: \"kubernetes.io/projected/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-kube-api-access-tqnbj\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015379 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015265 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-sys\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015379 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-root\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015379 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015350 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-root\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015576 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015428 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-sys\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015576 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.015576 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-textfile\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015576 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.015741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.015741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015628 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-metrics-client-ca\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015651 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0979b33-ec33-4fb0-8e9f-043991d0e998-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.015922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015855 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.015922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2tj\" (UniqueName: \"kubernetes.io/projected/a0979b33-ec33-4fb0-8e9f-043991d0e998-kube-api-access-2m2tj\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.016086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015919 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-textfile\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.016086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.015976 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016079 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016104 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdp7g\" (UniqueName: \"kubernetes.io/projected/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-api-access-fdp7g\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016129 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-wtmp\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.016323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.016323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016205 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.016323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016217 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-metrics-client-ca\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.016584 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016361 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-wtmp\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.016584 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:50:22.016444 2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 04:50:22.016584 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:50:22.016509 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls podName:e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a nodeName:}" failed. No retries permitted until 2026-09-16 04:50:22.516489571 +0000 UTC m=+88.030041098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls") pod "node-exporter-tqf6r" (UID: "e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a") : secret "node-exporter-tls" not found Sep 16 04:50:22.016759 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.016602 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0979b33-ec33-4fb0-8e9f-043991d0e998-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.016890 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:50:22.016871 2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 04:50:22.016967 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:50:22.016955 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls podName:8b61f1cf-bad9-45db-b9ac-5c4a67851a75 nodeName:}" failed. No retries permitted until 2026-09-16 04:50:22.516934294 +0000 UTC m=+88.030485810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-q62kf" (UID: "8b61f1cf-bad9-45db-b9ac-5c4a67851a75") : secret "kube-state-metrics-tls" not found Sep 16 04:50:22.017229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.017210 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.017297 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.017226 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.017492 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.017458 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.018188 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.018160 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.018281 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.018250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.018444 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.018410 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0979b33-ec33-4fb0-8e9f-043991d0e998-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.019169 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.019152 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.023761 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.023741 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnbj\" (UniqueName: \"kubernetes.io/projected/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-kube-api-access-tqnbj\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.028230 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.028211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2tj\" (UniqueName: \"kubernetes.io/projected/a0979b33-ec33-4fb0-8e9f-043991d0e998-kube-api-access-2m2tj\") pod \"openshift-state-metrics-67dcf8c979-pkbdz\" (UID: \"a0979b33-ec33-4fb0-8e9f-043991d0e998\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.028381 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.028340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdp7g\" (UniqueName: \"kubernetes.io/projected/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-api-access-fdp7g\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.108668 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.108643 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" Sep 16 04:50:22.379536 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.379513 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz"] Sep 16 04:50:22.381887 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:22.381861 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0979b33_ec33_4fb0_8e9f_043991d0e998.slice/crio-01fc78533b59d760acdd5bef633c98cba8c74bed9c60af75207f5957c9ab41ed WatchSource:0}: Error finding container 01fc78533b59d760acdd5bef633c98cba8c74bed9c60af75207f5957c9ab41ed: Status 404 returned error can't find the container with id 01fc78533b59d760acdd5bef633c98cba8c74bed9c60af75207f5957c9ab41ed Sep 16 04:50:22.520924 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.520893 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.521060 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.520953 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.523122 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.523100 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a-node-exporter-tls\") pod \"node-exporter-tqf6r\" (UID: \"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a\") " pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.523250 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.523228 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8b61f1cf-bad9-45db-b9ac-5c4a67851a75-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-q62kf\" (UID: \"8b61f1cf-bad9-45db-b9ac-5c4a67851a75\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.711184 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.711109 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" event={"ID":"a0979b33-ec33-4fb0-8e9f-043991d0e998","Type":"ContainerStarted","Data":"dccb376edaf1cc9e7f31d5837a62ceceb736dca1b001fb71461831aa3ebe7fd3"} Sep 16 04:50:22.711184 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.711169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" event={"ID":"a0979b33-ec33-4fb0-8e9f-043991d0e998","Type":"ContainerStarted","Data":"5ea7ade7b86d0ec638da6579046d86d64e345596874ea0a11c2e59943c8eb060"} Sep 16 04:50:22.711346 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.711188 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" event={"ID":"a0979b33-ec33-4fb0-8e9f-043991d0e998","Type":"ContainerStarted","Data":"01fc78533b59d760acdd5bef633c98cba8c74bed9c60af75207f5957c9ab41ed"} Sep 16 04:50:22.725384 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.725355 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" Sep 16 04:50:22.730480 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:22.730453 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tqf6r" Sep 16 04:50:22.741230 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:22.741207 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99aa397_ad3f_4fef_a8ad_c633f1b7ab4a.slice/crio-d6564016af5a93118de93fcf8924f1de3a9a1dfb32c6c1b32fa37d3b2b85bebb WatchSource:0}: Error finding container d6564016af5a93118de93fcf8924f1de3a9a1dfb32c6c1b32fa37d3b2b85bebb: Status 404 returned error can't find the container with id d6564016af5a93118de93fcf8924f1de3a9a1dfb32c6c1b32fa37d3b2b85bebb Sep 16 04:50:23.001207 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:23.001185 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf"] Sep 16 04:50:23.003637 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:23.003609 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b61f1cf_bad9_45db_b9ac_5c4a67851a75.slice/crio-3dee35c15c19178ec4988aaa4ce4b5503c9847a0c60dd6c2e068fd7ac25676b7 WatchSource:0}: Error finding container 3dee35c15c19178ec4988aaa4ce4b5503c9847a0c60dd6c2e068fd7ac25676b7: Status 404 returned error can't find the container with id 3dee35c15c19178ec4988aaa4ce4b5503c9847a0c60dd6c2e068fd7ac25676b7 Sep 16 04:50:23.718169 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:23.718125 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqf6r" event={"ID":"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a","Type":"ContainerStarted","Data":"d6564016af5a93118de93fcf8924f1de3a9a1dfb32c6c1b32fa37d3b2b85bebb"} Sep 16 04:50:23.719563 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:23.719521 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" event={"ID":"8b61f1cf-bad9-45db-b9ac-5c4a67851a75","Type":"ContainerStarted","Data":"3dee35c15c19178ec4988aaa4ce4b5503c9847a0c60dd6c2e068fd7ac25676b7"} Sep 16 04:50:24.638748 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.638722 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:24.725687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.725590 2560 generic.go:356] "Generic (PLEG): container finished" podID="e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a" containerID="ab24d44f7b3f4c591c0a54530c25b7639360b0eab496e0fe37a2c22688c9fa6d" exitCode=0 Sep 16 04:50:24.725687 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.725668 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqf6r" event={"ID":"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a","Type":"ContainerDied","Data":"ab24d44f7b3f4c591c0a54530c25b7639360b0eab496e0fe37a2c22688c9fa6d"} Sep 16 04:50:24.728298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.728271 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" event={"ID":"8b61f1cf-bad9-45db-b9ac-5c4a67851a75","Type":"ContainerStarted","Data":"b857defb4a8efaffb575c42ad2b2a72d3ea51c90b498d489dd1ab1941feccc49"} Sep 16 04:50:24.728431 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.728326 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" event={"ID":"8b61f1cf-bad9-45db-b9ac-5c4a67851a75","Type":"ContainerStarted","Data":"0875b85c06aec733e99b0f874a7965866f298968a44de846e59cd824e48e1ca3"} Sep 16 04:50:24.728431 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.728347 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" event={"ID":"8b61f1cf-bad9-45db-b9ac-5c4a67851a75","Type":"ContainerStarted","Data":"23f662d4947bc7e5a3ed597a01bd67c33b546e13449d2353e3010e667eae00e5"} Sep 16 04:50:24.730599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.730574 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" event={"ID":"a0979b33-ec33-4fb0-8e9f-043991d0e998","Type":"ContainerStarted","Data":"5c4a7b584fb6a229497ef72a8a14e5f748ae4f795e85ad6f0fcdc85ad7ff9e29"} Sep 16 04:50:24.759926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.759881 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-q62kf" podStartSLOduration=2.572029 podStartE2EDuration="3.759865371s" totalImagesPullingTime="1.187836371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:24.759227172 +0000 UTC m=+90.272778740" watchObservedRunningTime="2026-09-16 04:50:24.759865371 +0000 UTC m=+90.273416905" Sep 16 04:50:24.777102 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:24.777009 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pkbdz" podStartSLOduration=2.094809907 podStartE2EDuration="3.776990971s" totalImagesPullingTime="1.682181064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:24.776479961 +0000 UTC m=+90.290031494" watchObservedRunningTime="2026-09-16 04:50:24.776990971 +0000 UTC m=+90.290542506" Sep 16 04:50:25.737294 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:25.737255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqf6r" event={"ID":"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a","Type":"ContainerStarted","Data":"d14a6693be687a9594efb83a99a0bd080131f6cf0dc7f9d89a968f746eda2e00"} Sep 16 04:50:25.737742 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:25.737311 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tqf6r" event={"ID":"e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a","Type":"ContainerStarted","Data":"fa178c13a6b1813c3c7e9489621c369226ad414e6458743be2d4669ec767e75d"} Sep 16 04:50:26.120955 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.120903 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tqf6r" podStartSLOduration=3.58602856 podStartE2EDuration="5.120884975s" totalImagesPullingTime="1.447619044s" totalInitContainerRuntime="87.237371ms" isStatefulPod=true podCreationTimestamp="2026-09-16 04:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:25.757894839 +0000 UTC m=+91.271446382" watchObservedRunningTime="2026-09-16 04:50:26.120884975 +0000 UTC m=+91.634436509" Sep 16 04:50:26.121571 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.121543 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp"] Sep 16 04:50:26.124752 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.124738 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.127488 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.127466 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gk5md\"" Sep 16 04:50:26.127614 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.127494 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 04:50:26.127614 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.127494 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 04:50:26.127614 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.127494 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 04:50:26.128504 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.128488 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 04:50:26.128568 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.128521 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8lmgaepflnba8\"" Sep 16 04:50:26.134045 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.134014 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp"] Sep 16 04:50:26.251309 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251278 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-client-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251323 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-metrics-server-audit-profiles\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251351 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5fb\" (UniqueName: \"kubernetes.io/projected/3d43a618-7716-4aec-8efa-30dbe024f733-kube-api-access-kf5fb\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251411 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-tls\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251440 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d43a618-7716-4aec-8efa-30dbe024f733-audit-log\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251766 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251507 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.251766 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.251566 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-client-certs\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.351957 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.351925 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-tls\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352098 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.351968 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d43a618-7716-4aec-8efa-30dbe024f733-audit-log\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352098 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352000 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352098 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-client-certs\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352098 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352069 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-client-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352273 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-metrics-server-audit-profiles\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352323 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352279 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5fb\" (UniqueName: \"kubernetes.io/projected/3d43a618-7716-4aec-8efa-30dbe024f733-kube-api-access-kf5fb\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352804 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352775 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.352934 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.352843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3d43a618-7716-4aec-8efa-30dbe024f733-audit-log\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.353296 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.353266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3d43a618-7716-4aec-8efa-30dbe024f733-metrics-server-audit-profiles\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.354605 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.354577 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-client-ca-bundle\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.354704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.354654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-tls\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.354704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.354658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3d43a618-7716-4aec-8efa-30dbe024f733-secret-metrics-server-client-certs\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.360963 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.360939 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5fb\" (UniqueName: \"kubernetes.io/projected/3d43a618-7716-4aec-8efa-30dbe024f733-kube-api-access-kf5fb\") pod \"metrics-server-5d46fd8d8f-pgbrp\" (UID: \"3d43a618-7716-4aec-8efa-30dbe024f733\") " pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.434999 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.434928 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:26.501738 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.501706 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj"] Sep 16 04:50:26.506267 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.506249 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.509389 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.509365 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 04:50:26.509692 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.509674 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bkk6v\"" Sep 16 04:50:26.514980 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.514961 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj"] Sep 16 04:50:26.655513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.655485 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/517a0a44-a36c-42cf-bc5e-12b71a8838c7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.655700 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.655568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb29v\" (UniqueName: \"kubernetes.io/projected/517a0a44-a36c-42cf-bc5e-12b71a8838c7-kube-api-access-rb29v\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.709140 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.709098 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp"] Sep 16 04:50:26.711478 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:26.711444 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d43a618_7716_4aec_8efa_30dbe024f733.slice/crio-dfcd22059309295bc44a4ab61594eb1bb707a229b22d019c97ff2eebfcdb62ca WatchSource:0}: Error finding container dfcd22059309295bc44a4ab61594eb1bb707a229b22d019c97ff2eebfcdb62ca: Status 404 returned error can't find the container with id dfcd22059309295bc44a4ab61594eb1bb707a229b22d019c97ff2eebfcdb62ca Sep 16 04:50:26.742484 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.742453 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" event={"ID":"3d43a618-7716-4aec-8efa-30dbe024f733","Type":"ContainerStarted","Data":"dfcd22059309295bc44a4ab61594eb1bb707a229b22d019c97ff2eebfcdb62ca"} Sep 16 04:50:26.756115 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.756088 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/517a0a44-a36c-42cf-bc5e-12b71a8838c7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.756253 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.756213 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb29v\" (UniqueName: \"kubernetes.io/projected/517a0a44-a36c-42cf-bc5e-12b71a8838c7-kube-api-access-rb29v\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.758438 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.758418 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/517a0a44-a36c-42cf-bc5e-12b71a8838c7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.763529 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.763513 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb29v\" (UniqueName: \"kubernetes.io/projected/517a0a44-a36c-42cf-bc5e-12b71a8838c7-kube-api-access-rb29v\") pod \"monitoring-plugin-5d85778b9d-fs7hj\" (UID: \"517a0a44-a36c-42cf-bc5e-12b71a8838c7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:26.817500 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:26.817479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:27.084974 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:27.084845 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj"] Sep 16 04:50:27.088877 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:27.088852 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517a0a44_a36c_42cf_bc5e_12b71a8838c7.slice/crio-6ec1ca75658c054bc18b031dc17699bd09ce5efbb5ce2ec7db3ab41c6e01221f WatchSource:0}: Error finding container 6ec1ca75658c054bc18b031dc17699bd09ce5efbb5ce2ec7db3ab41c6e01221f: Status 404 returned error can't find the container with id 6ec1ca75658c054bc18b031dc17699bd09ce5efbb5ce2ec7db3ab41c6e01221f Sep 16 04:50:27.748691 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:27.748651 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" event={"ID":"517a0a44-a36c-42cf-bc5e-12b71a8838c7","Type":"ContainerStarted","Data":"6ec1ca75658c054bc18b031dc17699bd09ce5efbb5ce2ec7db3ab41c6e01221f"} Sep 16 04:50:28.270619 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.270581 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:50:28.754751 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.754712 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" event={"ID":"3d43a618-7716-4aec-8efa-30dbe024f733","Type":"ContainerStarted","Data":"1b8f021cbf52fdf8aafde27633b0c648a26d98eb794a29b53c9b69735f96cb89"} Sep 16 04:50:28.756270 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.756243 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" event={"ID":"517a0a44-a36c-42cf-bc5e-12b71a8838c7","Type":"ContainerStarted","Data":"fb53c4e5edec6dd1e61ceed511e613c6fd51f444499a860d522712f828bee39b"} Sep 16 04:50:28.756475 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.756458 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:28.761071 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.761051 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" Sep 16 04:50:28.771746 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.771709 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" podStartSLOduration=1.097376994 podStartE2EDuration="2.771695429s" totalImagesPullingTime="1.674318435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:28.770216719 +0000 UTC m=+94.283768278" watchObservedRunningTime="2026-09-16 04:50:28.771695429 +0000 UTC m=+94.285246962" Sep 16 04:50:28.786815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:28.786778 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs7hj" podStartSLOduration=1.486374691 podStartE2EDuration="2.786760819s" totalImagesPullingTime="1.300386128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:28.785731444 +0000 UTC m=+94.299282978" watchObservedRunningTime="2026-09-16 04:50:28.786760819 +0000 UTC m=+94.300312394" Sep 16 04:50:30.264298 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.264264 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:50:30.270038 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.270018 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.273026 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.273002 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 04:50:30.273168 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.273105 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 04:50:30.274108 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274088 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 04:50:30.274173 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274107 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pkcgd\"" Sep 16 04:50:30.274173 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274160 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 04:50:30.274423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274406 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 04:50:30.274497 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274412 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 04:50:30.274497 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.274413 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 04:50:30.280586 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.280567 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 04:50:30.285161 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.285138 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:50:30.384613 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384568 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384613 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384611 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384629 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxvl\" (UniqueName: \"kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384786 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384770 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.384914 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.384828 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485210 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485174 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485210 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485209 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485244 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxvl\" (UniqueName: \"kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485277 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485372 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485661 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485432 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.485997 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.485968 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.486108 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.486012 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.486210 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.486187 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.486301 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.486283 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.487792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.487774 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.487912 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.487894 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.495676 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.495653 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxvl\" (UniqueName: \"kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl\") pod \"console-5d46fbc6cb-mvdcq\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.579707 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.579679 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:30.852678 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:30.852652 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:50:30.855036 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:50:30.855014 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50dbdaf_68e6_4097_9366_ddf3c78bb362.slice/crio-74f91541f835d8f2d99ab148fdc921086f9b7ec694a34d9434568ebd6e2089fc WatchSource:0}: Error finding container 74f91541f835d8f2d99ab148fdc921086f9b7ec694a34d9434568ebd6e2089fc: Status 404 returned error can't find the container with id 74f91541f835d8f2d99ab148fdc921086f9b7ec694a34d9434568ebd6e2089fc Sep 16 04:50:31.770043 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:31.769912 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d46fbc6cb-mvdcq" event={"ID":"e50dbdaf-68e6-4097-9366-ddf3c78bb362","Type":"ContainerStarted","Data":"74f91541f835d8f2d99ab148fdc921086f9b7ec694a34d9434568ebd6e2089fc"} Sep 16 04:50:32.629762 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:32.629721 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v92kt" Sep 16 04:50:33.780729 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:33.780648 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d46fbc6cb-mvdcq" event={"ID":"e50dbdaf-68e6-4097-9366-ddf3c78bb362","Type":"ContainerStarted","Data":"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9"} Sep 16 04:50:33.800469 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:33.800419 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d46fbc6cb-mvdcq" podStartSLOduration=1.211654995 podStartE2EDuration="3.800401513s" totalImagesPullingTime="2.588746518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:50:33.798358164 +0000 UTC m=+99.311909723" watchObservedRunningTime="2026-09-16 04:50:33.800401513 +0000 UTC m=+99.313953047" Sep 16 04:50:40.580110 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:40.580070 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:40.580604 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:40.580135 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:40.584947 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:40.584919 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:40.810337 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:40.810305 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:50:44.829748 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:44.829717 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6074ca0-4d90-47b2-8c5e-069b750b9fd9" containerID="1c5f4c20f4451889b0878768f418a9eb70732c06229805e6b00614e931560e20" exitCode=0 Sep 16 04:50:44.830152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:44.829772 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" event={"ID":"b6074ca0-4d90-47b2-8c5e-069b750b9fd9","Type":"ContainerDied","Data":"1c5f4c20f4451889b0878768f418a9eb70732c06229805e6b00614e931560e20"} Sep 16 04:50:44.830152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:44.830050 2560 scope.go:117] "RemoveContainer" containerID="1c5f4c20f4451889b0878768f418a9eb70732c06229805e6b00614e931560e20" Sep 16 04:50:45.840345 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:45.840314 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-qf8dt" event={"ID":"b6074ca0-4d90-47b2-8c5e-069b750b9fd9","Type":"ContainerStarted","Data":"5b9c6faa5e690711e0ec834fdb4ab356466472f2930859a4144b05bb0a52cb4e"} Sep 16 04:50:46.435165 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:46.435133 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:46.435319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:46.435200 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:50:53.297205 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.297164 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" podUID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" containerName="registry" containerID="cri-o://63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04" gracePeriod=30 Sep 16 04:50:53.547980 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.547926 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:53.582297 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582269 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582335 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582359 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582386 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582413 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582711 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582462 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcvzr\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582711 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582518 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration\") pod \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\" (UID: \"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca\") " Sep 16 04:50:53.582825 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582795 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:50:53.582889 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.582820 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:50:53.585592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.585563 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:50:53.585728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.585702 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr" (OuterVolumeSpecName: "kube-api-access-fcvzr") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "kube-api-access-fcvzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:50:53.585887 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.585864 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:50:53.585972 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.585897 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:50:53.586105 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.586069 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:50:53.594227 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.594203 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" (UID: "93a70d1f-afc4-40d6-b8de-bbc5f98de9ca"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:50:53.683152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683121 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-bound-sa-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683149 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-ca-trust-extracted\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683152 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683158 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-trusted-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683167 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-certificates\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683175 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-installation-pull-secrets\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683184 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcvzr\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-kube-api-access-fcvzr\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683194 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-image-registry-private-configuration\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.683375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.683202 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca-registry-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:50:53.870399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.870368 2560 generic.go:356] "Generic (PLEG): container finished" podID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" containerID="63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04" exitCode=0 Sep 16 04:50:53.870592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.870414 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" Sep 16 04:50:53.870592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.870460 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" event={"ID":"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca","Type":"ContainerDied","Data":"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04"} Sep 16 04:50:53.870592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.870525 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f67679f5d-pk5v2" event={"ID":"93a70d1f-afc4-40d6-b8de-bbc5f98de9ca","Type":"ContainerDied","Data":"8435e34dbdab3822a8b3837b1838a8385e3e5cab8394c0489fa19117b02953b1"} Sep 16 04:50:53.870592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.870569 2560 scope.go:117] "RemoveContainer" containerID="63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04" Sep 16 04:50:53.897660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.896824 2560 scope.go:117] "RemoveContainer" containerID="63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04" Sep 16 04:50:53.898823 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:50:53.897976 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04\": container with ID starting with 63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04 not found: ID does not exist" containerID="63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04" Sep 16 04:50:53.898823 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.898018 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04"} err="failed to get container status \"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04\": rpc error: code = NotFound desc = could not find container \"63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04\": container with ID starting with 63b7c0d2ff41e0c03b0063a7d58203a2af0050acd16b5e700eabfad4ae781d04 not found: ID does not exist" Sep 16 04:50:53.910213 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.910186 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:50:53.913867 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:53.913845 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f67679f5d-pk5v2"] Sep 16 04:50:55.119692 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:50:55.119661 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" path="/var/lib/kubelet/pods/93a70d1f-afc4-40d6-b8de-bbc5f98de9ca/volumes" Sep 16 04:51:06.440854 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:06.440821 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:51:06.444456 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:06.444431 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d46fd8d8f-pgbrp" Sep 16 04:51:09.933824 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:09.933746 2560 generic.go:356] "Generic (PLEG): container finished" podID="31d4c6fc-9a85-49bd-b99d-bd839996e36d" containerID="c5df1d5d86ede83b76bbea889177dd201312bf494a0f0c387e2f32111f1bfda3" exitCode=0 Sep 16 04:51:09.933824 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:09.933791 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9l49f" event={"ID":"31d4c6fc-9a85-49bd-b99d-bd839996e36d","Type":"ContainerDied","Data":"c5df1d5d86ede83b76bbea889177dd201312bf494a0f0c387e2f32111f1bfda3"} Sep 16 04:51:09.934281 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:09.934169 2560 scope.go:117] "RemoveContainer" containerID="c5df1d5d86ede83b76bbea889177dd201312bf494a0f0c387e2f32111f1bfda3" Sep 16 04:51:10.939928 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:10.939897 2560 generic.go:356] "Generic (PLEG): container finished" podID="bc3f3f78-7e08-4046-8046-b8e0842d1258" containerID="4187c06f7ba27789e578166fdf0e3b1c9e3be43dbf286a1be68daf3cc519e6e8" exitCode=0 Sep 16 04:51:10.940373 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:10.939977 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" event={"ID":"bc3f3f78-7e08-4046-8046-b8e0842d1258","Type":"ContainerDied","Data":"4187c06f7ba27789e578166fdf0e3b1c9e3be43dbf286a1be68daf3cc519e6e8"} Sep 16 04:51:10.940373 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:10.940345 2560 scope.go:117] "RemoveContainer" containerID="4187c06f7ba27789e578166fdf0e3b1c9e3be43dbf286a1be68daf3cc519e6e8" Sep 16 04:51:10.942054 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:10.942030 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9l49f" event={"ID":"31d4c6fc-9a85-49bd-b99d-bd839996e36d","Type":"ContainerStarted","Data":"6c77acb81e9641d031b37981281dafb03fa82b032389fa918591e15af8bdfa16"} Sep 16 04:51:11.948105 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:11.948070 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-t8lzj" event={"ID":"bc3f3f78-7e08-4046-8046-b8e0842d1258","Type":"ContainerStarted","Data":"b5fbb6b367cda06b3d76b0b594b5240f9e78030f55274651ee32ded81e65244b"} Sep 16 04:51:46.064163 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.064128 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr"] Sep 16 04:51:46.064638 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.064491 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" containerName="registry" Sep 16 04:51:46.064638 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.064521 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" containerName="registry" Sep 16 04:51:46.064752 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.064641 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a70d1f-afc4-40d6-b8de-bbc5f98de9ca" containerName="registry" Sep 16 04:51:46.069174 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.069153 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.071745 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.071711 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 04:51:46.071745 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.071735 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t7zqt\"" Sep 16 04:51:46.071902 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.071742 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 04:51:46.072045 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.072015 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 04:51:46.072170 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.072105 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 04:51:46.072170 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.072137 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 04:51:46.077534 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.077118 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 04:51:46.084562 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.084524 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr"] Sep 16 04:51:46.200380 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200341 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200380 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200382 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200443 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445lm\" (UniqueName: \"kubernetes.io/projected/92d1a55b-289a-49a8-b612-3f9e4c1c806d-kube-api-access-445lm\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200499 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-metrics-client-ca\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200524 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-serving-certs-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200541 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.200617 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.200595 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-federate-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301317 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301282 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-federate-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301328 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301399 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445lm\" (UniqueName: \"kubernetes.io/projected/92d1a55b-289a-49a8-b612-3f9e4c1c806d-kube-api-access-445lm\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301434 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301423 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-metrics-client-ca\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301665 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301451 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-serving-certs-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.301665 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.301606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.302278 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.302250 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-metrics-client-ca\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.302278 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.302266 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-serving-certs-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.302627 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.302602 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.303978 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.303950 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.304111 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.304088 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-telemeter-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.304214 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.304199 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-federate-client-tls\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.304251 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.304205 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92d1a55b-289a-49a8-b612-3f9e4c1c806d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.308917 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.308895 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445lm\" (UniqueName: \"kubernetes.io/projected/92d1a55b-289a-49a8-b612-3f9e4c1c806d-kube-api-access-445lm\") pod \"telemeter-client-74fdb4b7fc-8sthr\" (UID: \"92d1a55b-289a-49a8-b612-3f9e4c1c806d\") " pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.383180 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.383151 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" Sep 16 04:51:46.662050 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:46.662025 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr"] Sep 16 04:51:46.664539 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:51:46.664513 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d1a55b_289a_49a8_b612_3f9e4c1c806d.slice/crio-75f6323c95715ba7f8b412f2068c2190a8a4a4befb0deb8de4f1c69c48679372 WatchSource:0}: Error finding container 75f6323c95715ba7f8b412f2068c2190a8a4a4befb0deb8de4f1c69c48679372: Status 404 returned error can't find the container with id 75f6323c95715ba7f8b412f2068c2190a8a4a4befb0deb8de4f1c69c48679372 Sep 16 04:51:47.078840 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:47.078785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" event={"ID":"92d1a55b-289a-49a8-b612-3f9e4c1c806d","Type":"ContainerStarted","Data":"75f6323c95715ba7f8b412f2068c2190a8a4a4befb0deb8de4f1c69c48679372"} Sep 16 04:51:49.090585 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:49.090535 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" event={"ID":"92d1a55b-289a-49a8-b612-3f9e4c1c806d","Type":"ContainerStarted","Data":"1a092a73f6aada48c5e61ae5f3e100877d6aef717d079de733f2d3dc54c5cdd1"} Sep 16 04:51:50.097092 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.097057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" event={"ID":"92d1a55b-289a-49a8-b612-3f9e4c1c806d","Type":"ContainerStarted","Data":"6853a86508c396322bb4df100c7c60ae0d19048ae0eb17216cabda9680f14f40"} Sep 16 04:51:50.097448 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.097123 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" event={"ID":"92d1a55b-289a-49a8-b612-3f9e4c1c806d","Type":"ContainerStarted","Data":"e6f615247701252d9c0754b6dad6b571a19dd6fea03a8e15729727b5e438c1e8"} Sep 16 04:51:50.125140 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.125083 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-74fdb4b7fc-8sthr" podStartSLOduration=1.4235459590000001 podStartE2EDuration="4.125067597s" totalImagesPullingTime="2.701521638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:51:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:51:50.118835764 +0000 UTC m=+175.632387300" watchObservedRunningTime="2026-09-16 04:51:50.125067597 +0000 UTC m=+175.638619130" Sep 16 04:51:50.192005 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.191975 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:51:50.195785 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.195764 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.207001 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.206979 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:51:50.329897 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.329853 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330052 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.329904 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330052 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.329961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330052 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.330024 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330154 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.330114 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330187 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.330155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr7c\" (UniqueName: \"kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.330187 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.330175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.430782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430695 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.430782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr7c\" (UniqueName: \"kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.430782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430768 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430789 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430838 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431080 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.430863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431585 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.431521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431747 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.431587 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431810 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.431675 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.431810 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.431680 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.433443 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.433421 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.433498 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.433428 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.438060 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.438041 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr7c\" (UniqueName: \"kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c\") pod \"console-6765746bc9-p6slx\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.506179 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.506151 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:51:50.780182 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:50.780157 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:51:50.782521 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:51:50.782490 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0664d8b_3772_424b_ac14_46e59bdc35dd.slice/crio-d323b57be2be4c2ad1a2fb50b0a839ad3c1cd5e38adfef4df14c6f6bf85f02fe WatchSource:0}: Error finding container d323b57be2be4c2ad1a2fb50b0a839ad3c1cd5e38adfef4df14c6f6bf85f02fe: Status 404 returned error can't find the container with id d323b57be2be4c2ad1a2fb50b0a839ad3c1cd5e38adfef4df14c6f6bf85f02fe Sep 16 04:51:51.102963 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:51.102931 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6765746bc9-p6slx" event={"ID":"a0664d8b-3772-424b-ac14-46e59bdc35dd","Type":"ContainerStarted","Data":"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa"} Sep 16 04:51:51.103352 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:51.102984 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6765746bc9-p6slx" event={"ID":"a0664d8b-3772-424b-ac14-46e59bdc35dd","Type":"ContainerStarted","Data":"d323b57be2be4c2ad1a2fb50b0a839ad3c1cd5e38adfef4df14c6f6bf85f02fe"} Sep 16 04:51:51.122731 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:51:51.122683 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6765746bc9-p6slx" podStartSLOduration=1.12266793 podStartE2EDuration="1.12266793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:51:51.121651384 +0000 UTC m=+176.635202940" watchObservedRunningTime="2026-09-16 04:51:51.12266793 +0000 UTC m=+176.636219464" Sep 16 04:52:00.506560 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:00.506525 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:52:00.506915 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:00.506618 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:52:00.511436 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:00.511412 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:52:01.152986 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:01.152958 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:52:01.190009 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:01.189970 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:52:26.212039 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.211999 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d46fbc6cb-mvdcq" podUID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" containerName="console" containerID="cri-o://a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9" gracePeriod=15 Sep 16 04:52:26.453685 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.453660 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:52:26.525009 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.524925 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525009 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.524967 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525009 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.524982 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525113 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525159 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525196 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhxvl\" (UniqueName: \"kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525236 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert\") pod \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\" (UID: \"e50dbdaf-68e6-4097-9366-ddf3c78bb362\") " Sep 16 04:52:26.525527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525313 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca" (OuterVolumeSpecName: "service-ca") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:52:26.525527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525361 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config" (OuterVolumeSpecName: "console-config") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:52:26.525527 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525376 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:52:26.525686 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525645 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.525686 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525666 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.525686 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525680 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-trusted-ca-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.525802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.525737 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:52:26.527428 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.527398 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:52:26.527533 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.527423 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:52:26.527533 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.527411 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl" (OuterVolumeSpecName: "kube-api-access-dhxvl") pod "e50dbdaf-68e6-4097-9366-ddf3c78bb362" (UID: "e50dbdaf-68e6-4097-9366-ddf3c78bb362"). InnerVolumeSpecName "kube-api-access-dhxvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:52:26.626472 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.626444 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.626472 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.626467 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhxvl\" (UniqueName: \"kubernetes.io/projected/e50dbdaf-68e6-4097-9366-ddf3c78bb362-kube-api-access-dhxvl\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.626472 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.626476 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e50dbdaf-68e6-4097-9366-ddf3c78bb362-oauth-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:26.626693 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:26.626484 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e50dbdaf-68e6-4097-9366-ddf3c78bb362-console-oauth-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:52:27.256995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.256957 2560 generic.go:356] "Generic (PLEG): container finished" podID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" containerID="a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9" exitCode=0 Sep 16 04:52:27.256995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.256999 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d46fbc6cb-mvdcq" event={"ID":"e50dbdaf-68e6-4097-9366-ddf3c78bb362","Type":"ContainerDied","Data":"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9"} Sep 16 04:52:27.257485 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.257004 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d46fbc6cb-mvdcq" Sep 16 04:52:27.257485 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.257047 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d46fbc6cb-mvdcq" event={"ID":"e50dbdaf-68e6-4097-9366-ddf3c78bb362","Type":"ContainerDied","Data":"74f91541f835d8f2d99ab148fdc921086f9b7ec694a34d9434568ebd6e2089fc"} Sep 16 04:52:27.257485 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.257075 2560 scope.go:117] "RemoveContainer" containerID="a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9" Sep 16 04:52:27.265191 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.265173 2560 scope.go:117] "RemoveContainer" containerID="a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9" Sep 16 04:52:27.265446 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:52:27.265428 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9\": container with ID starting with a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9 not found: ID does not exist" containerID="a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9" Sep 16 04:52:27.265510 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.265459 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9"} err="failed to get container status \"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9\": rpc error: code = NotFound desc = could not find container \"a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9\": container with ID starting with a508647794f94f417fb524301a364acdf05537eeb894c053fb632d7906aff0f9 not found: ID does not exist" Sep 16 04:52:27.274150 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.274128 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:52:27.278353 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:27.278331 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d46fbc6cb-mvdcq"] Sep 16 04:52:29.120234 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:52:29.120200 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" path="/var/lib/kubelet/pods/e50dbdaf-68e6-4097-9366-ddf3c78bb362/volumes" Sep 16 04:53:07.563927 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.563894 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:53:07.564374 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.564205 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" containerName="console" Sep 16 04:53:07.564374 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.564217 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" containerName="console" Sep 16 04:53:07.564374 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.564282 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50dbdaf-68e6-4097-9366-ddf3c78bb362" containerName="console" Sep 16 04:53:07.568558 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.568525 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.577736 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.577715 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:53:07.652002 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.651978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652005 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652025 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652043 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652064 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppq4\" (UniqueName: \"kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.652294 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.652168 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753401 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753374 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753415 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753441 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753468 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753495 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753521 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753518 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.753801 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.753541 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppq4\" (UniqueName: \"kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.754279 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.754257 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.754391 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.754334 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.754498 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.754475 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.754579 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.754485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.755967 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.755938 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.756083 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.756061 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.761731 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.761708 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppq4\" (UniqueName: \"kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4\") pod \"console-88df9b4f6-6cb62\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:07.879591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:07.879539 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:08.156387 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:08.156363 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:53:08.158931 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:53:08.158907 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbade65e4_cfc1_4c4f_910e_e7ea98c025aa.slice/crio-59277a9f0cd2c900158ede5c55dcff70740870c7d08f419646876e3b1fc8758e WatchSource:0}: Error finding container 59277a9f0cd2c900158ede5c55dcff70740870c7d08f419646876e3b1fc8758e: Status 404 returned error can't find the container with id 59277a9f0cd2c900158ede5c55dcff70740870c7d08f419646876e3b1fc8758e Sep 16 04:53:08.430405 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:08.430309 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df9b4f6-6cb62" event={"ID":"bade65e4-cfc1-4c4f-910e-e7ea98c025aa","Type":"ContainerStarted","Data":"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998"} Sep 16 04:53:08.430405 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:08.430397 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df9b4f6-6cb62" event={"ID":"bade65e4-cfc1-4c4f-910e-e7ea98c025aa","Type":"ContainerStarted","Data":"59277a9f0cd2c900158ede5c55dcff70740870c7d08f419646876e3b1fc8758e"} Sep 16 04:53:08.448676 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:08.448635 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-88df9b4f6-6cb62" podStartSLOduration=1.448619716 podStartE2EDuration="1.448619716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:53:08.44735474 +0000 UTC m=+253.960906278" watchObservedRunningTime="2026-09-16 04:53:08.448619716 +0000 UTC m=+253.962171249" Sep 16 04:53:17.879749 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:17.879673 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:17.879749 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:17.879739 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:17.884592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:17.884568 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:18.476163 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:18.476137 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:53:18.522290 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:18.522260 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:53:43.543371 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.543310 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6765746bc9-p6slx" podUID="a0664d8b-3772-424b-ac14-46e59bdc35dd" containerName="console" containerID="cri-o://591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa" gracePeriod=15 Sep 16 04:53:43.783318 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.783289 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:53:43.834980 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.834955 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835093 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.834998 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835093 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835022 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835093 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835049 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835093 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835076 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835093 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835093 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcr7c\" (UniqueName: \"kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835337 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835135 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle\") pod \"a0664d8b-3772-424b-ac14-46e59bdc35dd\" (UID: \"a0664d8b-3772-424b-ac14-46e59bdc35dd\") " Sep 16 04:53:43.835602 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835505 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:53:43.835713 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835619 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:53:43.835713 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835630 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:53:43.835713 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.835689 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config" (OuterVolumeSpecName: "console-config") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:53:43.837083 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.837055 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:53:43.837582 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.837540 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c" (OuterVolumeSpecName: "kube-api-access-bcr7c") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "kube-api-access-bcr7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:53:43.837683 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.837540 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a0664d8b-3772-424b-ac14-46e59bdc35dd" (UID: "a0664d8b-3772-424b-ac14-46e59bdc35dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:53:43.935836 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935814 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935836 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935834 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-oauth-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935959 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935845 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935959 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935854 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr7c\" (UniqueName: \"kubernetes.io/projected/a0664d8b-3772-424b-ac14-46e59bdc35dd-kube-api-access-bcr7c\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935959 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935862 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-trusted-ca-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935959 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935890 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-oauth-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:43.935959 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:43.935899 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0664d8b-3772-424b-ac14-46e59bdc35dd-console-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:53:44.576728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.576695 2560 generic.go:356] "Generic (PLEG): container finished" podID="a0664d8b-3772-424b-ac14-46e59bdc35dd" containerID="591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa" exitCode=0 Sep 16 04:53:44.577197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.576746 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6765746bc9-p6slx" Sep 16 04:53:44.577197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.576767 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6765746bc9-p6slx" event={"ID":"a0664d8b-3772-424b-ac14-46e59bdc35dd","Type":"ContainerDied","Data":"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa"} Sep 16 04:53:44.577197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.576806 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6765746bc9-p6slx" event={"ID":"a0664d8b-3772-424b-ac14-46e59bdc35dd","Type":"ContainerDied","Data":"d323b57be2be4c2ad1a2fb50b0a839ad3c1cd5e38adfef4df14c6f6bf85f02fe"} Sep 16 04:53:44.577197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.576827 2560 scope.go:117] "RemoveContainer" containerID="591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa" Sep 16 04:53:44.585639 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.585620 2560 scope.go:117] "RemoveContainer" containerID="591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa" Sep 16 04:53:44.585903 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:53:44.585881 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa\": container with ID starting with 591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa not found: ID does not exist" containerID="591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa" Sep 16 04:53:44.585970 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.585915 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa"} err="failed to get container status \"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa\": rpc error: code = NotFound desc = could not find container \"591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa\": container with ID starting with 591119b0116b6c54be213127e8d0ca1491811ddfaa1393d428ac10aee00ae1aa not found: ID does not exist" Sep 16 04:53:44.596985 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.596962 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:53:44.600384 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:44.600366 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6765746bc9-p6slx"] Sep 16 04:53:45.119719 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:45.119692 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0664d8b-3772-424b-ac14-46e59bdc35dd" path="/var/lib/kubelet/pods/a0664d8b-3772-424b-ac14-46e59bdc35dd/volumes" Sep 16 04:53:54.905784 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:54.905760 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:53:54.909028 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:54.909006 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:53:54.913083 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:54.913059 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:53:54.916335 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:54.916316 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:53:54.916838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:53:54.916822 2560 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 04:54:31.758854 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.758819 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f"] Sep 16 04:54:31.761399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.759307 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0664d8b-3772-424b-ac14-46e59bdc35dd" containerName="console" Sep 16 04:54:31.761399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.759324 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0664d8b-3772-424b-ac14-46e59bdc35dd" containerName="console" Sep 16 04:54:31.761399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.759387 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0664d8b-3772-424b-ac14-46e59bdc35dd" containerName="console" Sep 16 04:54:31.762415 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.762393 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.765091 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.765071 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 04:54:31.765195 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.765118 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 04:54:31.766166 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.766148 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hbjz8\"" Sep 16 04:54:31.771269 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.771249 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f"] Sep 16 04:54:31.833406 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.833382 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.833505 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.833419 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.833559 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.833523 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t7k\" (UniqueName: \"kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.934825 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.934801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t7k\" (UniqueName: \"kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.934947 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.934844 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.934947 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.934869 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.935307 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.935280 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.935345 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.935303 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:31.944076 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:31.944054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t7k\" (UniqueName: \"kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:32.072697 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:32.072633 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:32.341714 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:32.341579 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f"] Sep 16 04:54:32.344266 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:54:32.344241 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203ea390_2c84_4586_80d3_5f629ca807c5.slice/crio-5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8 WatchSource:0}: Error finding container 5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8: Status 404 returned error can't find the container with id 5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8 Sep 16 04:54:32.346227 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:32.346212 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 04:54:32.769118 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:32.769032 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerStarted","Data":"5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8"} Sep 16 04:54:38.799191 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:38.799156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerStarted","Data":"e62209d35180b02e0e8bcb0cf760131df0c5bbae589dfa8263f112e41d110b88"} Sep 16 04:54:39.804951 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:39.804919 2560 generic.go:356] "Generic (PLEG): container finished" podID="203ea390-2c84-4586-80d3-5f629ca807c5" containerID="e62209d35180b02e0e8bcb0cf760131df0c5bbae589dfa8263f112e41d110b88" exitCode=0 Sep 16 04:54:39.805319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:39.804982 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerDied","Data":"e62209d35180b02e0e8bcb0cf760131df0c5bbae589dfa8263f112e41d110b88"} Sep 16 04:54:42.820921 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:42.820883 2560 generic.go:356] "Generic (PLEG): container finished" podID="203ea390-2c84-4586-80d3-5f629ca807c5" containerID="c51cfe16fcff77a4c98672f28661378333676e9ed91614318378576034a9f1e1" exitCode=0 Sep 16 04:54:42.821398 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:42.820944 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerDied","Data":"c51cfe16fcff77a4c98672f28661378333676e9ed91614318378576034a9f1e1"} Sep 16 04:54:49.855627 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:49.855593 2560 generic.go:356] "Generic (PLEG): container finished" podID="203ea390-2c84-4586-80d3-5f629ca807c5" containerID="f999026c26d71ca11cf87e1f926ebd7c57320797729cbcae25618f8a4692cb02" exitCode=0 Sep 16 04:54:49.855985 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:49.855650 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerDied","Data":"f999026c26d71ca11cf87e1f926ebd7c57320797729cbcae25618f8a4692cb02"} Sep 16 04:54:50.983326 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:50.983295 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:51.106222 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.106186 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util\") pod \"203ea390-2c84-4586-80d3-5f629ca807c5\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " Sep 16 04:54:51.106399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.106233 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle\") pod \"203ea390-2c84-4586-80d3-5f629ca807c5\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " Sep 16 04:54:51.106399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.106274 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2t7k\" (UniqueName: \"kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k\") pod \"203ea390-2c84-4586-80d3-5f629ca807c5\" (UID: \"203ea390-2c84-4586-80d3-5f629ca807c5\") " Sep 16 04:54:51.106929 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.106901 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle" (OuterVolumeSpecName: "bundle") pod "203ea390-2c84-4586-80d3-5f629ca807c5" (UID: "203ea390-2c84-4586-80d3-5f629ca807c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:54:51.108453 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.108418 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k" (OuterVolumeSpecName: "kube-api-access-r2t7k") pod "203ea390-2c84-4586-80d3-5f629ca807c5" (UID: "203ea390-2c84-4586-80d3-5f629ca807c5"). InnerVolumeSpecName "kube-api-access-r2t7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:54:51.111287 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.111234 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util" (OuterVolumeSpecName: "util") pod "203ea390-2c84-4586-80d3-5f629ca807c5" (UID: "203ea390-2c84-4586-80d3-5f629ca807c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:54:51.207208 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.207168 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t7k\" (UniqueName: \"kubernetes.io/projected/203ea390-2c84-4586-80d3-5f629ca807c5-kube-api-access-r2t7k\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:54:51.207208 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.207201 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:54:51.207208 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.207216 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/203ea390-2c84-4586-80d3-5f629ca807c5-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:54:51.866623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.866589 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" event={"ID":"203ea390-2c84-4586-80d3-5f629ca807c5","Type":"ContainerDied","Data":"5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8"} Sep 16 04:54:51.866623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.866617 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gl46f" Sep 16 04:54:51.866835 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:51.866637 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e56c1b894f00f88a48d031f87d8b300550780f6f3c289e7620928c41290cab8" Sep 16 04:54:59.765422 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765394 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd"] Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765702 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="extract" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765715 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="extract" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765725 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="pull" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765730 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="pull" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765743 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="util" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765749 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="util" Sep 16 04:54:59.765850 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.765819 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="203ea390-2c84-4586-80d3-5f629ca807c5" containerName="extract" Sep 16 04:54:59.770755 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.770736 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.773306 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.773281 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 16 04:54:59.773306 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.773298 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:54:59.773466 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.773373 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9g5fv\"" Sep 16 04:54:59.782838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.782817 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd"] Sep 16 04:54:59.869731 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.869704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a4fe9f-7760-413b-9125-c8197369d57f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.869731 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.869740 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9xj\" (UniqueName: \"kubernetes.io/projected/85a4fe9f-7760-413b-9125-c8197369d57f-kube-api-access-hl9xj\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.970363 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.970329 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a4fe9f-7760-413b-9125-c8197369d57f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.970363 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.970374 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9xj\" (UniqueName: \"kubernetes.io/projected/85a4fe9f-7760-413b-9125-c8197369d57f-kube-api-access-hl9xj\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.970771 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.970751 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85a4fe9f-7760-413b-9125-c8197369d57f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:54:59.981161 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:54:59.981138 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9xj\" (UniqueName: \"kubernetes.io/projected/85a4fe9f-7760-413b-9125-c8197369d57f-kube-api-access-hl9xj\") pod \"cert-manager-operator-controller-manager-55c444bf86-76crd\" (UID: \"85a4fe9f-7760-413b-9125-c8197369d57f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:55:00.080149 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:00.080125 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" Sep 16 04:55:00.365631 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:00.365604 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd"] Sep 16 04:55:00.368797 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:00.368767 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a4fe9f_7760_413b_9125_c8197369d57f.slice/crio-4c1499d54f682f951509e7cae65d28f5ef6ab8575dc206d56840096eaa31a430 WatchSource:0}: Error finding container 4c1499d54f682f951509e7cae65d28f5ef6ab8575dc206d56840096eaa31a430: Status 404 returned error can't find the container with id 4c1499d54f682f951509e7cae65d28f5ef6ab8575dc206d56840096eaa31a430 Sep 16 04:55:00.907663 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:00.907622 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" event={"ID":"85a4fe9f-7760-413b-9125-c8197369d57f","Type":"ContainerStarted","Data":"4c1499d54f682f951509e7cae65d28f5ef6ab8575dc206d56840096eaa31a430"} Sep 16 04:55:02.917597 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:02.917563 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" event={"ID":"85a4fe9f-7760-413b-9125-c8197369d57f","Type":"ContainerStarted","Data":"95920e3df37827ff1a9726f852cbb1f1abbade4a4aaed0a333d4b4f4f789ac5f"} Sep 16 04:55:02.941445 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:02.941400 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-76crd" podStartSLOduration=2.410854878 podStartE2EDuration="3.941384384s" totalImagesPullingTime="1.530529506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:54:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:02.939078339 +0000 UTC m=+368.452629874" watchObservedRunningTime="2026-09-16 04:55:02.941384384 +0000 UTC m=+368.454935918" Sep 16 04:55:03.766040 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.766008 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr"] Sep 16 04:55:03.769592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.769575 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.772295 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.772273 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 04:55:03.772531 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.772513 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hbjz8\"" Sep 16 04:55:03.774094 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.774079 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 04:55:03.785008 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.784986 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr"] Sep 16 04:55:03.808287 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.808261 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.808378 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.808294 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgrl\" (UniqueName: \"kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.808378 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.808324 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.909425 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.909394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.909597 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.909433 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgrl\" (UniqueName: \"kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.909597 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.909467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.909840 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.909820 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.909901 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.909856 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:03.918198 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:03.918177 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgrl\" (UniqueName: \"kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:04.078715 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:04.078695 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:04.355908 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:04.355881 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr"] Sep 16 04:55:04.358265 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:04.358230 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac509c11_2d4d_4333_a3b2_05d263cfbbef.slice/crio-5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba WatchSource:0}: Error finding container 5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba: Status 404 returned error can't find the container with id 5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba Sep 16 04:55:04.927352 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:04.927313 2560 generic.go:356] "Generic (PLEG): container finished" podID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerID="61abbc56f90506e60770ffd8dee42d1544801f958c1a6907af507a072c3f3e8e" exitCode=0 Sep 16 04:55:04.927707 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:04.927390 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" event={"ID":"ac509c11-2d4d-4333-a3b2-05d263cfbbef","Type":"ContainerDied","Data":"61abbc56f90506e60770ffd8dee42d1544801f958c1a6907af507a072c3f3e8e"} Sep 16 04:55:04.927707 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:04.927470 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" event={"ID":"ac509c11-2d4d-4333-a3b2-05d263cfbbef","Type":"ContainerStarted","Data":"5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba"} Sep 16 04:55:07.041962 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.041937 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7drrz"] Sep 16 04:55:07.049299 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.049280 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.052089 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.052072 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hcdt7\"" Sep 16 04:55:07.052178 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.052125 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 16 04:55:07.052233 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.052201 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 16 04:55:07.055607 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.055587 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7drrz"] Sep 16 04:55:07.136801 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.136777 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.136904 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.136852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2cp\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-kube-api-access-8n2cp\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.237775 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.237743 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2cp\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-kube-api-access-8n2cp\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.237918 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.237790 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.247750 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.247725 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2cp\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-kube-api-access-8n2cp\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.247909 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.247889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1ec964b5-616e-4e34-8ae4-888e035075e3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7drrz\" (UID: \"1ec964b5-616e-4e34-8ae4-888e035075e3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.374590 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.374566 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" Sep 16 04:55:07.644998 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.644973 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7drrz"] Sep 16 04:55:07.647447 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:07.647424 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ec964b5_616e_4e34_8ae4_888e035075e3.slice/crio-af1a4c04af9bda7432b801631b6c952b823c360b09cb0d14c17ed80905ddc9cc WatchSource:0}: Error finding container af1a4c04af9bda7432b801631b6c952b823c360b09cb0d14c17ed80905ddc9cc: Status 404 returned error can't find the container with id af1a4c04af9bda7432b801631b6c952b823c360b09cb0d14c17ed80905ddc9cc Sep 16 04:55:07.944695 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.944600 2560 generic.go:356] "Generic (PLEG): container finished" podID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerID="ad80b1dbdf8771d102c3f18787a5b674b16221b86553e54110cc058faa1d7d65" exitCode=0 Sep 16 04:55:07.944855 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.944707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" event={"ID":"ac509c11-2d4d-4333-a3b2-05d263cfbbef","Type":"ContainerDied","Data":"ad80b1dbdf8771d102c3f18787a5b674b16221b86553e54110cc058faa1d7d65"} Sep 16 04:55:07.946067 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:07.945979 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" event={"ID":"1ec964b5-616e-4e34-8ae4-888e035075e3","Type":"ContainerStarted","Data":"af1a4c04af9bda7432b801631b6c952b823c360b09cb0d14c17ed80905ddc9cc"} Sep 16 04:55:08.954409 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:08.954374 2560 generic.go:356] "Generic (PLEG): container finished" podID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerID="34defb4921e9eb61d343af5c59097f1aa53b0a792067f2e896095cd5279f6ac0" exitCode=0 Sep 16 04:55:08.954883 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:08.954467 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" event={"ID":"ac509c11-2d4d-4333-a3b2-05d263cfbbef","Type":"ContainerDied","Data":"34defb4921e9eb61d343af5c59097f1aa53b0a792067f2e896095cd5279f6ac0"} Sep 16 04:55:10.220315 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.220264 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:10.264059 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.264026 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbgrl\" (UniqueName: \"kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl\") pod \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " Sep 16 04:55:10.264193 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.264108 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util\") pod \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " Sep 16 04:55:10.264193 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.264170 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle\") pod \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\" (UID: \"ac509c11-2d4d-4333-a3b2-05d263cfbbef\") " Sep 16 04:55:10.264653 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.264625 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle" (OuterVolumeSpecName: "bundle") pod "ac509c11-2d4d-4333-a3b2-05d263cfbbef" (UID: "ac509c11-2d4d-4333-a3b2-05d263cfbbef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:10.266169 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.266147 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl" (OuterVolumeSpecName: "kube-api-access-cbgrl") pod "ac509c11-2d4d-4333-a3b2-05d263cfbbef" (UID: "ac509c11-2d4d-4333-a3b2-05d263cfbbef"). InnerVolumeSpecName "kube-api-access-cbgrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:55:10.268231 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.268200 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util" (OuterVolumeSpecName: "util") pod "ac509c11-2d4d-4333-a3b2-05d263cfbbef" (UID: "ac509c11-2d4d-4333-a3b2-05d263cfbbef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:10.365902 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.365873 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:10.365902 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.365897 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac509c11-2d4d-4333-a3b2-05d263cfbbef-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:10.365902 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.365907 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgrl\" (UniqueName: \"kubernetes.io/projected/ac509c11-2d4d-4333-a3b2-05d263cfbbef-kube-api-access-cbgrl\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:10.965615 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.965586 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" Sep 16 04:55:10.965809 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.965584 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnxqr" event={"ID":"ac509c11-2d4d-4333-a3b2-05d263cfbbef","Type":"ContainerDied","Data":"5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba"} Sep 16 04:55:10.965809 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.965677 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5496482f635d76b30182da03d16cce609ded376e59353ee9170da4e375fed8ba" Sep 16 04:55:10.967192 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.967160 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" event={"ID":"1ec964b5-616e-4e34-8ae4-888e035075e3","Type":"ContainerStarted","Data":"47c357865a8354018a6826ecd1b7e37dfff8bf2a791296d671875e83a9250575"} Sep 16 04:55:10.986668 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:10.986523 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7drrz" podStartSLOduration=1.3672194260000001 podStartE2EDuration="3.986505184s" totalImagesPullingTime="2.619285758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:10.985891412 +0000 UTC m=+376.499442947" watchObservedRunningTime="2026-09-16 04:55:10.986505184 +0000 UTC m=+376.500056724" Sep 16 04:55:11.041889 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.041859 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nr4sg"] Sep 16 04:55:11.042171 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042159 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="pull" Sep 16 04:55:11.042229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042172 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="pull" Sep 16 04:55:11.042229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042184 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="extract" Sep 16 04:55:11.042229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042189 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="extract" Sep 16 04:55:11.042229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042196 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="util" Sep 16 04:55:11.042229 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042202 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="util" Sep 16 04:55:11.042399 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.042258 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac509c11-2d4d-4333-a3b2-05d263cfbbef" containerName="extract" Sep 16 04:55:11.045511 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.045493 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.048730 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.048708 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gplt8\"" Sep 16 04:55:11.055963 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.055939 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nr4sg"] Sep 16 04:55:11.172508 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.172472 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmj5\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-kube-api-access-2pmj5\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.172698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.172527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.273375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.273290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmj5\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-kube-api-access-2pmj5\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.273375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.273343 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.281371 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.281343 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.281509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.281490 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmj5\" (UniqueName: \"kubernetes.io/projected/6e8d848c-e2f3-4b72-aa9c-2fc051112239-kube-api-access-2pmj5\") pod \"cert-manager-webhook-6486ffd997-nr4sg\" (UID: \"6e8d848c-e2f3-4b72-aa9c-2fc051112239\") " pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.356678 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.356652 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.629537 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.629514 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-nr4sg"] Sep 16 04:55:11.631683 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:11.631656 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8d848c_e2f3_4b72_aa9c_2fc051112239.slice/crio-1aef6498bd5d1285ba2dae346b77f13c69d02b546ba6c50b38b14bf5e5befaa3 WatchSource:0}: Error finding container 1aef6498bd5d1285ba2dae346b77f13c69d02b546ba6c50b38b14bf5e5befaa3: Status 404 returned error can't find the container with id 1aef6498bd5d1285ba2dae346b77f13c69d02b546ba6c50b38b14bf5e5befaa3 Sep 16 04:55:11.973599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.973502 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" event={"ID":"6e8d848c-e2f3-4b72-aa9c-2fc051112239","Type":"ContainerStarted","Data":"a04cafebdb5a9ab208a42ddebe16375ae1d71217074a1966bce6d624ff6d0361"} Sep 16 04:55:11.973599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.973558 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" event={"ID":"6e8d848c-e2f3-4b72-aa9c-2fc051112239","Type":"ContainerStarted","Data":"1aef6498bd5d1285ba2dae346b77f13c69d02b546ba6c50b38b14bf5e5befaa3"} Sep 16 04:55:11.973599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.973582 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:11.992658 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:11.992615 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" podStartSLOduration=0.992599939 podStartE2EDuration="992.599939ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:11.99073389 +0000 UTC m=+377.504285437" watchObservedRunningTime="2026-09-16 04:55:11.992599939 +0000 UTC m=+377.506151473" Sep 16 04:55:17.981052 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:17.981020 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-nr4sg" Sep 16 04:55:25.045348 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.045314 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw"] Sep 16 04:55:25.050821 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.050789 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.053344 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.053323 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 04:55:25.053460 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.053419 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 04:55:25.054483 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.054456 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hbjz8\"" Sep 16 04:55:25.056606 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.056584 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw"] Sep 16 04:55:25.187197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.187162 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.187393 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.187310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.187393 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.187353 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6dw\" (UniqueName: \"kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.288646 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.288613 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.288802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.288699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.288802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.288722 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6dw\" (UniqueName: \"kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.289013 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.288992 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.289054 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.289016 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.296591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.296526 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6dw\" (UniqueName: \"kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.362319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.362293 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:25.636589 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:25.636564 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw"] Sep 16 04:55:25.638969 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:25.638938 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46f8bbd_6c02_4cb7_896b_48eda69b370c.slice/crio-1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025 WatchSource:0}: Error finding container 1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025: Status 404 returned error can't find the container with id 1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025 Sep 16 04:55:26.040664 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:26.040580 2560 generic.go:356] "Generic (PLEG): container finished" podID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerID="0ece8e4622ea27fc4ed1f53d42535e473c73dec78f3d3966d4fa39775bda74dd" exitCode=0 Sep 16 04:55:26.040823 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:26.040669 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" event={"ID":"d46f8bbd-6c02-4cb7-896b-48eda69b370c","Type":"ContainerDied","Data":"0ece8e4622ea27fc4ed1f53d42535e473c73dec78f3d3966d4fa39775bda74dd"} Sep 16 04:55:26.040823 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:26.040730 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" event={"ID":"d46f8bbd-6c02-4cb7-896b-48eda69b370c","Type":"ContainerStarted","Data":"1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025"} Sep 16 04:55:27.047417 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:27.047327 2560 generic.go:356] "Generic (PLEG): container finished" podID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerID="87237f0d4be975a2b9df425f62ada7563a4323aa18cc411b43f8ca5ca209845d" exitCode=0 Sep 16 04:55:27.047417 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:27.047400 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" event={"ID":"d46f8bbd-6c02-4cb7-896b-48eda69b370c","Type":"ContainerDied","Data":"87237f0d4be975a2b9df425f62ada7563a4323aa18cc411b43f8ca5ca209845d"} Sep 16 04:55:28.054054 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:28.054021 2560 generic.go:356] "Generic (PLEG): container finished" podID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerID="98361cde745728227f35f1ec1a9becf4c3508251ed6ba891dea46413db8aa71c" exitCode=0 Sep 16 04:55:28.054429 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:28.054095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" event={"ID":"d46f8bbd-6c02-4cb7-896b-48eda69b370c","Type":"ContainerDied","Data":"98361cde745728227f35f1ec1a9becf4c3508251ed6ba891dea46413db8aa71c"} Sep 16 04:55:29.185750 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.185723 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:29.327890 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.327860 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk6dw\" (UniqueName: \"kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw\") pod \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " Sep 16 04:55:29.328068 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.327904 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util\") pod \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " Sep 16 04:55:29.328068 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.327956 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle\") pod \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\" (UID: \"d46f8bbd-6c02-4cb7-896b-48eda69b370c\") " Sep 16 04:55:29.328728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.328684 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle" (OuterVolumeSpecName: "bundle") pod "d46f8bbd-6c02-4cb7-896b-48eda69b370c" (UID: "d46f8bbd-6c02-4cb7-896b-48eda69b370c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:29.330140 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.330113 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw" (OuterVolumeSpecName: "kube-api-access-rk6dw") pod "d46f8bbd-6c02-4cb7-896b-48eda69b370c" (UID: "d46f8bbd-6c02-4cb7-896b-48eda69b370c"). InnerVolumeSpecName "kube-api-access-rk6dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:55:29.333430 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.333407 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util" (OuterVolumeSpecName: "util") pod "d46f8bbd-6c02-4cb7-896b-48eda69b370c" (UID: "d46f8bbd-6c02-4cb7-896b-48eda69b370c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:29.429490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.429451 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6dw\" (UniqueName: \"kubernetes.io/projected/d46f8bbd-6c02-4cb7-896b-48eda69b370c-kube-api-access-rk6dw\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:29.429490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.429485 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:29.429490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:29.429495 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d46f8bbd-6c02-4cb7-896b-48eda69b370c-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:30.070067 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:30.070044 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" Sep 16 04:55:30.070067 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:30.070043 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afld6tw" event={"ID":"d46f8bbd-6c02-4cb7-896b-48eda69b370c","Type":"ContainerDied","Data":"1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025"} Sep 16 04:55:30.070279 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:30.070089 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1740220c858fd87db87786fb53fdd4fffa80c656e6863c0b702824953c225025" Sep 16 04:55:35.535621 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.535586 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54956d744f-2jn8f"] Sep 16 04:55:35.536057 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536040 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="extract" Sep 16 04:55:35.536130 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536059 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="extract" Sep 16 04:55:35.536130 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536072 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="util" Sep 16 04:55:35.536130 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536080 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="util" Sep 16 04:55:35.536130 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536101 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="pull" Sep 16 04:55:35.536130 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536107 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="pull" Sep 16 04:55:35.536320 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.536181 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d46f8bbd-6c02-4cb7-896b-48eda69b370c" containerName="extract" Sep 16 04:55:35.539491 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.539474 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.552736 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.552712 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54956d744f-2jn8f"] Sep 16 04:55:35.684226 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684191 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-oauth-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684226 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684233 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684252 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-console-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684274 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-service-ca\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684308 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-oauth-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684412 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684349 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46s7w\" (UniqueName: \"kubernetes.io/projected/7f259120-d360-468d-87a6-4ab7e01864a5-kube-api-access-46s7w\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.684541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.684413 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-trusted-ca-bundle\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785172 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-oauth-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-console-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-service-ca\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785319 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-oauth-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785531 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785484 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46s7w\" (UniqueName: \"kubernetes.io/projected/7f259120-d360-468d-87a6-4ab7e01864a5-kube-api-access-46s7w\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.785682 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.785633 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-trusted-ca-bundle\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.786104 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.786082 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-console-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.786104 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.786093 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-service-ca\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.786278 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.786135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-oauth-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.786533 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.786510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f259120-d360-468d-87a6-4ab7e01864a5-trusted-ca-bundle\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.787775 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.787752 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-oauth-config\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.787894 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.787878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f259120-d360-468d-87a6-4ab7e01864a5-console-serving-cert\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.793848 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.793827 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46s7w\" (UniqueName: \"kubernetes.io/projected/7f259120-d360-468d-87a6-4ab7e01864a5-kube-api-access-46s7w\") pod \"console-54956d744f-2jn8f\" (UID: \"7f259120-d360-468d-87a6-4ab7e01864a5\") " pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:35.849214 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:35.849183 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:36.041135 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.041053 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp"] Sep 16 04:55:36.045796 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.045780 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.049589 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.049562 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 04:55:36.049690 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.049672 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hbjz8\"" Sep 16 04:55:36.050673 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.050655 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 04:55:36.068765 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.068739 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp"] Sep 16 04:55:36.129824 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.129799 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54956d744f-2jn8f"] Sep 16 04:55:36.131535 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:36.131502 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f259120_d360_468d_87a6_4ab7e01864a5.slice/crio-af0bf702af7cdc9eb323e7c36b93af7d16938398456f0ba6da21c89b373fdfac WatchSource:0}: Error finding container af0bf702af7cdc9eb323e7c36b93af7d16938398456f0ba6da21c89b373fdfac: Status 404 returned error can't find the container with id af0bf702af7cdc9eb323e7c36b93af7d16938398456f0ba6da21c89b373fdfac Sep 16 04:55:36.189047 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.189021 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.189151 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.189121 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.189213 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.189147 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2wdt\" (UniqueName: \"kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.290326 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.290293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.290477 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.290404 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.290477 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.290438 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2wdt\" (UniqueName: \"kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.290782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.290762 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.290830 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.290777 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.306004 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.305948 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2wdt\" (UniqueName: \"kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.355311 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.355265 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:36.644139 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:36.644110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp"] Sep 16 04:55:36.645859 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:36.645830 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d66ae1_d486_4f14_aaab_188a8f8e538f.slice/crio-9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2 WatchSource:0}: Error finding container 9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2: Status 404 returned error can't find the container with id 9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2 Sep 16 04:55:37.103462 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.103425 2560 generic.go:356] "Generic (PLEG): container finished" podID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerID="b4fac7c67f092fb5ada46ae30111ee1bbe40d1bbd83ba7a55cd95f9615e7d097" exitCode=0 Sep 16 04:55:37.103660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.103511 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" event={"ID":"57d66ae1-d486-4f14-aaab-188a8f8e538f","Type":"ContainerDied","Data":"b4fac7c67f092fb5ada46ae30111ee1bbe40d1bbd83ba7a55cd95f9615e7d097"} Sep 16 04:55:37.103660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.103589 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" event={"ID":"57d66ae1-d486-4f14-aaab-188a8f8e538f","Type":"ContainerStarted","Data":"9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2"} Sep 16 04:55:37.105267 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.105240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54956d744f-2jn8f" event={"ID":"7f259120-d360-468d-87a6-4ab7e01864a5","Type":"ContainerStarted","Data":"39381d52bc2647cb4351d985b09de7f56c8e5708ece4c9b3706950c51d69d5a3"} Sep 16 04:55:37.105392 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.105287 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54956d744f-2jn8f" event={"ID":"7f259120-d360-468d-87a6-4ab7e01864a5","Type":"ContainerStarted","Data":"af0bf702af7cdc9eb323e7c36b93af7d16938398456f0ba6da21c89b373fdfac"} Sep 16 04:55:37.147970 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.147913 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54956d744f-2jn8f" podStartSLOduration=2.147894428 podStartE2EDuration="2.147894428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:37.147125023 +0000 UTC m=+402.660676581" watchObservedRunningTime="2026-09-16 04:55:37.147894428 +0000 UTC m=+402.661445961" Sep 16 04:55:37.813749 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.813719 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd"] Sep 16 04:55:37.817351 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.817334 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:37.820584 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.820561 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 04:55:37.820696 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.820607 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 04:55:37.820787 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.820758 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 04:55:37.821000 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.820982 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zn4sz\"" Sep 16 04:55:37.821078 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.821008 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 04:55:37.831029 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.831006 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd"] Sep 16 04:55:37.905633 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.905602 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:37.905633 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.905633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7xr\" (UniqueName: \"kubernetes.io/projected/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-kube-api-access-5w7xr\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:37.905838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.905660 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:37.905838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.905689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:37.905838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:37.905740 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.006753 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.006665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.006930 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.006760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.006930 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.006791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7xr\" (UniqueName: \"kubernetes.io/projected/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-kube-api-access-5w7xr\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.006930 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.006819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.006930 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.006856 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.007161 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.007103 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-manifests\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.009053 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.009028 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-tmp\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.009300 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.009280 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.009423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.009405 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-webhook-cert\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.014328 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.014311 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7xr\" (UniqueName: \"kubernetes.io/projected/0cbf593d-cef9-4a7f-8dd1-86fdf372811a-kube-api-access-5w7xr\") pod \"opendatahub-operator-controller-manager-756589c867-zbnrd\" (UID: \"0cbf593d-cef9-4a7f-8dd1-86fdf372811a\") " pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.112591 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.112527 2560 generic.go:356] "Generic (PLEG): container finished" podID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerID="dba4fcb743f85014413bf0f599bd339cbb0d900cfc22ccdfa8909d60befec8bd" exitCode=0 Sep 16 04:55:38.112730 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.112608 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" event={"ID":"57d66ae1-d486-4f14-aaab-188a8f8e538f","Type":"ContainerDied","Data":"dba4fcb743f85014413bf0f599bd339cbb0d900cfc22ccdfa8909d60befec8bd"} Sep 16 04:55:38.192234 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.192209 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:38.483769 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:38.483747 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd"] Sep 16 04:55:38.485768 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:38.485740 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbf593d_cef9_4a7f_8dd1_86fdf372811a.slice/crio-949f16ad823ea7c750eaeebf9e046bc1cce98c6cf3a907c2e32c908ebad3e6ca WatchSource:0}: Error finding container 949f16ad823ea7c750eaeebf9e046bc1cce98c6cf3a907c2e32c908ebad3e6ca: Status 404 returned error can't find the container with id 949f16ad823ea7c750eaeebf9e046bc1cce98c6cf3a907c2e32c908ebad3e6ca Sep 16 04:55:39.129963 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:39.124761 2560 generic.go:356] "Generic (PLEG): container finished" podID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerID="651a086119b1a90ef16dcfe8588cdc25adddf7ec1488a83e361c472e25d549f5" exitCode=0 Sep 16 04:55:39.129963 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:39.124918 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" event={"ID":"57d66ae1-d486-4f14-aaab-188a8f8e538f","Type":"ContainerDied","Data":"651a086119b1a90ef16dcfe8588cdc25adddf7ec1488a83e361c472e25d549f5"} Sep 16 04:55:39.132010 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:39.131978 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" event={"ID":"0cbf593d-cef9-4a7f-8dd1-86fdf372811a","Type":"ContainerStarted","Data":"949f16ad823ea7c750eaeebf9e046bc1cce98c6cf3a907c2e32c908ebad3e6ca"} Sep 16 04:55:40.848040 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:40.848009 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:41.035074 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.035038 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2wdt\" (UniqueName: \"kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt\") pod \"57d66ae1-d486-4f14-aaab-188a8f8e538f\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " Sep 16 04:55:41.035236 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.035106 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util\") pod \"57d66ae1-d486-4f14-aaab-188a8f8e538f\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " Sep 16 04:55:41.035236 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.035196 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle\") pod \"57d66ae1-d486-4f14-aaab-188a8f8e538f\" (UID: \"57d66ae1-d486-4f14-aaab-188a8f8e538f\") " Sep 16 04:55:41.036027 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.035997 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle" (OuterVolumeSpecName: "bundle") pod "57d66ae1-d486-4f14-aaab-188a8f8e538f" (UID: "57d66ae1-d486-4f14-aaab-188a8f8e538f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:41.037563 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.037526 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt" (OuterVolumeSpecName: "kube-api-access-g2wdt") pod "57d66ae1-d486-4f14-aaab-188a8f8e538f" (UID: "57d66ae1-d486-4f14-aaab-188a8f8e538f"). InnerVolumeSpecName "kube-api-access-g2wdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:55:41.041240 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.041191 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util" (OuterVolumeSpecName: "util") pod "57d66ae1-d486-4f14-aaab-188a8f8e538f" (UID: "57d66ae1-d486-4f14-aaab-188a8f8e538f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:55:41.136822 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.136796 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2wdt\" (UniqueName: \"kubernetes.io/projected/57d66ae1-d486-4f14-aaab-188a8f8e538f-kube-api-access-g2wdt\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:41.136822 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.136820 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:41.136993 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.136830 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/57d66ae1-d486-4f14-aaab-188a8f8e538f-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:55:41.146914 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.146887 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" Sep 16 04:55:41.147043 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.146887 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0xf7gp" event={"ID":"57d66ae1-d486-4f14-aaab-188a8f8e538f","Type":"ContainerDied","Data":"9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2"} Sep 16 04:55:41.147043 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:41.147007 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9abad4d1dbeae7331ce5c65f0d9c05a9dd33f40590bd321854beffbd6b683ff2" Sep 16 04:55:43.160375 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:43.160342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" event={"ID":"0cbf593d-cef9-4a7f-8dd1-86fdf372811a","Type":"ContainerStarted","Data":"e9d49ebe3a70d80c81370b8dcb6f86a0f4d8bd3d37a76375094b2201957193d6"} Sep 16 04:55:44.166760 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:44.166722 2560 generic.go:356] "Generic (PLEG): container finished" podID="0cbf593d-cef9-4a7f-8dd1-86fdf372811a" containerID="e9d49ebe3a70d80c81370b8dcb6f86a0f4d8bd3d37a76375094b2201957193d6" exitCode=0 Sep 16 04:55:44.167136 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:44.166804 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" event={"ID":"0cbf593d-cef9-4a7f-8dd1-86fdf372811a","Type":"ContainerDied","Data":"e9d49ebe3a70d80c81370b8dcb6f86a0f4d8bd3d37a76375094b2201957193d6"} Sep 16 04:55:45.173384 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.173355 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" event={"ID":"0cbf593d-cef9-4a7f-8dd1-86fdf372811a","Type":"ContainerStarted","Data":"a863304dbafd040555fba6b146483aaa8f54ddef85a690dfac0057e065c067c5"} Sep 16 04:55:45.173872 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.173499 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:45.195306 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.195258 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" podStartSLOduration=3.174889445 podStartE2EDuration="8.195212824s" totalImagesPullingTime="3.956893428s" totalInitContainerRuntime="1.063429951s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:55:45.193086725 +0000 UTC m=+410.706638259" watchObservedRunningTime="2026-09-16 04:55:45.195212824 +0000 UTC m=+410.708764359" Sep 16 04:55:45.850042 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.849994 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:45.850233 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.850076 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:45.855088 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:45.855070 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:46.181865 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:46.181786 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54956d744f-2jn8f" Sep 16 04:55:46.228341 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:46.228304 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:55:56.180599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:56.180571 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-756589c867-zbnrd" Sep 16 04:55:58.251423 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251389 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl"] Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251714 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="extract" Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251728 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="extract" Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251752 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="util" Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251757 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="util" Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251766 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="pull" Sep 16 04:55:58.251792 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251771 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="pull" Sep 16 04:55:58.251977 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.251824 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57d66ae1-d486-4f14-aaab-188a8f8e538f" containerName="extract" Sep 16 04:55:58.256275 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.256254 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.259623 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.259605 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dnwl7\"" Sep 16 04:55:58.259736 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.259620 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 04:55:58.259736 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.259637 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 04:55:58.269962 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.269938 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl"] Sep 16 04:55:58.377453 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.377420 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tmp\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.377684 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.377497 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vscxq\" (UniqueName: \"kubernetes.io/projected/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-kube-api-access-vscxq\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.377684 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.377614 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tls-certs\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.478347 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.478309 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vscxq\" (UniqueName: \"kubernetes.io/projected/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-kube-api-access-vscxq\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.478535 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.478369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tls-certs\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.478535 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.478398 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tmp\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.480705 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.480673 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tmp\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.480838 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.480820 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-tls-certs\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.486324 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.486302 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vscxq\" (UniqueName: \"kubernetes.io/projected/495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde-kube-api-access-vscxq\") pod \"kube-auth-proxy-6965d5747b-6b9hl\" (UID: \"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde\") " pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.565418 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.565338 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" Sep 16 04:55:58.852247 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:58.852220 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl"] Sep 16 04:55:58.854323 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:55:58.854297 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495fa8f0_2ec0_4dd3_837d_8b3f1da8ccde.slice/crio-77318ef1ad621f972fc39fb81b252bc48c1dd1519d6f0214a6bc9316baa15ed6 WatchSource:0}: Error finding container 77318ef1ad621f972fc39fb81b252bc48c1dd1519d6f0214a6bc9316baa15ed6: Status 404 returned error can't find the container with id 77318ef1ad621f972fc39fb81b252bc48c1dd1519d6f0214a6bc9316baa15ed6 Sep 16 04:55:59.241101 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.241021 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" event={"ID":"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde","Type":"ContainerStarted","Data":"77318ef1ad621f972fc39fb81b252bc48c1dd1519d6f0214a6bc9316baa15ed6"} Sep 16 04:55:59.527841 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.527766 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:55:59.532373 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.532353 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.538866 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.538841 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 16 04:55:59.538995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.538867 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 04:55:59.538995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.538872 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 16 04:55:59.538995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.538915 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vx458\"" Sep 16 04:55:59.538995 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.538839 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 16 04:55:59.546510 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.546486 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:55:59.589628 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589601 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589746 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589662 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589746 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589688 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589746 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589884 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589884 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.589884 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.589801 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbwb\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690328 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690281 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690328 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690426 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690449 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbwb\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.690698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.690526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.691482 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.691433 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.693807 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.693758 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.694704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.694654 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.694813 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.694770 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.694919 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.694896 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.699449 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.699421 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.700012 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.699968 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbwb\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb\") pod \"istiod-openshift-gateway-7b7995d5cd-lch5w\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:55:59.848165 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:55:59.848128 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:56:00.148520 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:00.148447 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:56:00.150867 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:56:00.150665 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a39843f_4c13_4d21_b25f_cac297e010dc.slice/crio-f4168b0e007bd70db43f836ecc37b3cdfd7b9e2944e3a1904fe6c8bfac3142c9 WatchSource:0}: Error finding container f4168b0e007bd70db43f836ecc37b3cdfd7b9e2944e3a1904fe6c8bfac3142c9: Status 404 returned error can't find the container with id f4168b0e007bd70db43f836ecc37b3cdfd7b9e2944e3a1904fe6c8bfac3142c9 Sep 16 04:56:00.249250 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:00.249210 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" event={"ID":"7a39843f-4c13-4d21-b25f-cac297e010dc","Type":"ContainerStarted","Data":"f4168b0e007bd70db43f836ecc37b3cdfd7b9e2944e3a1904fe6c8bfac3142c9"} Sep 16 04:56:02.267079 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:02.266987 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" event={"ID":"495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde","Type":"ContainerStarted","Data":"3c0500d0b3cffe5620347e0e9acbacba981a55b29ced50f365999b2e1a3e3688"} Sep 16 04:56:02.312420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:02.312352 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6965d5747b-6b9hl" podStartSLOduration=1.318388198 podStartE2EDuration="4.312329324s" totalImagesPullingTime="2.993941126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:02.309150289 +0000 UTC m=+427.822701822" watchObservedRunningTime="2026-09-16 04:56:02.312329324 +0000 UTC m=+427.825880865" Sep 16 04:56:03.627865 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:03.627821 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 04:56:03.628145 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:03.627889 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 04:56:04.281164 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:04.281125 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" event={"ID":"7a39843f-4c13-4d21-b25f-cac297e010dc","Type":"ContainerStarted","Data":"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c"} Sep 16 04:56:04.281344 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:04.281316 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:56:04.282490 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:04.282463 2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lch5w container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 04:56:04.282636 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:04.282517 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 04:56:04.304419 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:04.304370 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" podStartSLOduration=1.8306505849999999 podStartE2EDuration="5.304356051s" totalImagesPullingTime="3.473705466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:04.303525417 +0000 UTC m=+429.817076974" watchObservedRunningTime="2026-09-16 04:56:04.304356051 +0000 UTC m=+429.817907586" Sep 16 04:56:05.286351 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:05.286326 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:56:11.256247 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.256187 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-88df9b4f6-6cb62" podUID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" containerName="console" containerID="cri-o://f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998" gracePeriod=15 Sep 16 04:56:11.496685 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.496661 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:56:11.604014 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.603983 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604185 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604030 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604185 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604088 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604185 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604107 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604185 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604143 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604394 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604191 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604394 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604215 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kppq4\" (UniqueName: \"kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4\") pod \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\" (UID: \"bade65e4-cfc1-4c4f-910e-e7ea98c025aa\") " Sep 16 04:56:11.604587 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604531 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config" (OuterVolumeSpecName: "console-config") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:56:11.604660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604578 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:56:11.604660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604538 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:56:11.604660 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.604615 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:56:11.606370 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.606342 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:56:11.606370 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.606357 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4" (OuterVolumeSpecName: "kube-api-access-kppq4") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "kube-api-access-kppq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:56:11.606524 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.606365 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bade65e4-cfc1-4c4f-910e-e7ea98c025aa" (UID: "bade65e4-cfc1-4c4f-910e-e7ea98c025aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:56:11.705745 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705718 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705745 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705739 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705745 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705748 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-trusted-ca-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705756 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-oauth-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705765 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppq4\" (UniqueName: \"kubernetes.io/projected/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-kube-api-access-kppq4\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705787 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-serving-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:11.705926 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:11.705796 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bade65e4-cfc1-4c4f-910e-e7ea98c025aa-console-oauth-config\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:12.320451 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.320412 2560 generic.go:356] "Generic (PLEG): container finished" podID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" containerID="f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998" exitCode=0 Sep 16 04:56:12.320853 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.320464 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-88df9b4f6-6cb62" Sep 16 04:56:12.320853 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.320478 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df9b4f6-6cb62" event={"ID":"bade65e4-cfc1-4c4f-910e-e7ea98c025aa","Type":"ContainerDied","Data":"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998"} Sep 16 04:56:12.320853 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.320528 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-88df9b4f6-6cb62" event={"ID":"bade65e4-cfc1-4c4f-910e-e7ea98c025aa","Type":"ContainerDied","Data":"59277a9f0cd2c900158ede5c55dcff70740870c7d08f419646876e3b1fc8758e"} Sep 16 04:56:12.320853 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.320571 2560 scope.go:117] "RemoveContainer" containerID="f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998" Sep 16 04:56:12.329734 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.329715 2560 scope.go:117] "RemoveContainer" containerID="f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998" Sep 16 04:56:12.329990 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:56:12.329971 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998\": container with ID starting with f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998 not found: ID does not exist" containerID="f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998" Sep 16 04:56:12.330033 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.330000 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998"} err="failed to get container status \"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998\": rpc error: code = NotFound desc = could not find container \"f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998\": container with ID starting with f9f8799726dffe23609fd8254a981902018f857f3e38c737526f9273cdc13998 not found: ID does not exist" Sep 16 04:56:12.347176 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.347150 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:56:12.348978 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:12.348960 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-88df9b4f6-6cb62"] Sep 16 04:56:13.121206 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:13.121173 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" path="/var/lib/kubelet/pods/bade65e4-cfc1-4c4f-910e-e7ea98c025aa/volumes" Sep 16 04:56:17.277943 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.277907 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq"] Sep 16 04:56:17.278471 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.278449 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" containerName="console" Sep 16 04:56:17.278582 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.278474 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" containerName="console" Sep 16 04:56:17.278645 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.278613 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bade65e4-cfc1-4c4f-910e-e7ea98c025aa" containerName="console" Sep 16 04:56:17.287862 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.287840 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.293073 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.293043 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 04:56:17.293172 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.293043 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 04:56:17.293172 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.293097 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-shcnf\"" Sep 16 04:56:17.293172 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.293146 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 04:56:17.293495 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.293475 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 04:56:17.297656 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.297640 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 04:56:17.305977 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.305954 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq"] Sep 16 04:56:17.454815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.454787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.454985 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.454829 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svh8q\" (UniqueName: \"kubernetes.io/projected/32e03a38-2b05-4bda-abf4-1d67b749d622-kube-api-access-svh8q\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.454985 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.454857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32e03a38-2b05-4bda-abf4-1d67b749d622-manager-config\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.454985 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.454872 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-metrics-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.556048 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.555916 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.556048 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.555981 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svh8q\" (UniqueName: \"kubernetes.io/projected/32e03a38-2b05-4bda-abf4-1d67b749d622-kube-api-access-svh8q\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.556048 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.556020 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32e03a38-2b05-4bda-abf4-1d67b749d622-manager-config\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.556048 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.556037 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-metrics-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.556960 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.556925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32e03a38-2b05-4bda-abf4-1d67b749d622-manager-config\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.558710 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.558683 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.558913 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.558893 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32e03a38-2b05-4bda-abf4-1d67b749d622-metrics-cert\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.567601 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.567573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svh8q\" (UniqueName: \"kubernetes.io/projected/32e03a38-2b05-4bda-abf4-1d67b749d622-kube-api-access-svh8q\") pod \"lws-controller-manager-778cf7776f-xtqcq\" (UID: \"32e03a38-2b05-4bda-abf4-1d67b749d622\") " pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.597109 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.597084 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:17.893862 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:17.893828 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq"] Sep 16 04:56:18.357414 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:18.357378 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" event={"ID":"32e03a38-2b05-4bda-abf4-1d67b749d622","Type":"ContainerStarted","Data":"09c3b54a5b2a86f37e8fe72d4e861f80ec00d79e9175bcc228aca1df3436d9bd"} Sep 16 04:56:20.369509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:20.369472 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" event={"ID":"32e03a38-2b05-4bda-abf4-1d67b749d622","Type":"ContainerStarted","Data":"9de6c13a0bbb7200cbb58d035aeefccf714f886357ae222327ca2101e4874542"} Sep 16 04:56:20.369896 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:20.369530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:56:20.409566 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:20.408840 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" podStartSLOduration=1.84213516 podStartE2EDuration="3.40882135s" totalImagesPullingTime="1.56668619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:56:20.405786401 +0000 UTC m=+445.919337939" watchObservedRunningTime="2026-09-16 04:56:20.40882135 +0000 UTC m=+445.922372884" Sep 16 04:56:22.481045 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.481015 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:56:22.481418 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.481191 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerName="discovery" containerID="cri-o://3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c" gracePeriod=30 Sep 16 04:56:22.729147 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.729125 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:56:22.821179 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821106 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821179 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821139 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltbwb\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821179 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821161 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821191 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821206 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821244 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821486 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap\") pod \"7a39843f-4c13-4d21-b25f-cac297e010dc\" (UID: \"7a39843f-4c13-4d21-b25f-cac297e010dc\") " Sep 16 04:56:22.821737 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.821715 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 04:56:22.823488 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823464 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:56:22.823637 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823585 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb" (OuterVolumeSpecName: "kube-api-access-ltbwb") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "kube-api-access-ltbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:56:22.823782 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823758 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts" (OuterVolumeSpecName: "cacerts") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:56:22.823835 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823770 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs" (OuterVolumeSpecName: "local-certs") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:56:22.823888 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823871 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 04:56:22.823955 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.823937 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a39843f-4c13-4d21-b25f-cac297e010dc" (UID: "7a39843f-4c13-4d21-b25f-cac297e010dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:56:22.922567 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922521 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-ca-configmap\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922567 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922568 2560 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-cacerts\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922579 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbwb\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-kube-api-access-ltbwb\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922590 2560 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-csr-dns-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922599 2560 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a39843f-4c13-4d21-b25f-cac297e010dc-local-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922607 2560 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-kubeconfig\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:22.922741 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:22.922615 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a39843f-4c13-4d21-b25f-cac297e010dc-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:56:23.386487 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.386398 2560 generic.go:356] "Generic (PLEG): container finished" podID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerID="3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c" exitCode=0 Sep 16 04:56:23.386487 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.386446 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" Sep 16 04:56:23.386704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.386480 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" event={"ID":"7a39843f-4c13-4d21-b25f-cac297e010dc","Type":"ContainerDied","Data":"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c"} Sep 16 04:56:23.386704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.386530 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w" event={"ID":"7a39843f-4c13-4d21-b25f-cac297e010dc","Type":"ContainerDied","Data":"f4168b0e007bd70db43f836ecc37b3cdfd7b9e2944e3a1904fe6c8bfac3142c9"} Sep 16 04:56:23.386704 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.386571 2560 scope.go:117] "RemoveContainer" containerID="3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c" Sep 16 04:56:23.395474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.395455 2560 scope.go:117] "RemoveContainer" containerID="3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c" Sep 16 04:56:23.395773 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:56:23.395753 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c\": container with ID starting with 3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c not found: ID does not exist" containerID="3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c" Sep 16 04:56:23.395869 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.395785 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c"} err="failed to get container status \"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c\": rpc error: code = NotFound desc = could not find container \"3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c\": container with ID starting with 3c0f402bcd3acab61298c5586d2e74d26a7083e44cc5da0e6ceb88dc618ac74c not found: ID does not exist" Sep 16 04:56:23.407290 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.407260 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:56:23.414641 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:23.414614 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lch5w"] Sep 16 04:56:25.120922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:25.120893 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" path="/var/lib/kubelet/pods/7a39843f-4c13-4d21-b25f-cac297e010dc/volumes" Sep 16 04:56:31.377228 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:56:31.377199 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-778cf7776f-xtqcq" Sep 16 04:57:13.726073 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.726040 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7"] Sep 16 04:57:13.726509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.726437 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerName="discovery" Sep 16 04:57:13.726509 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.726454 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerName="discovery" Sep 16 04:57:13.726599 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.726588 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a39843f-4c13-4d21-b25f-cac297e010dc" containerName="discovery" Sep 16 04:57:13.731070 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.731054 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.733905 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.733885 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 04:57:13.733905 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.733896 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 04:57:13.734071 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.733888 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hbjz8\"" Sep 16 04:57:13.740251 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.740228 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7"] Sep 16 04:57:13.822069 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.822031 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn"] Sep 16 04:57:13.825757 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.825734 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:13.835134 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.835112 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn"] Sep 16 04:57:13.849425 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.849389 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.849657 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.849624 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmmn\" (UniqueName: \"kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.849802 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.849780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.924920 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.924888 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq"] Sep 16 04:57:13.928381 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.928359 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:13.938155 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.938118 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq"] Sep 16 04:57:13.950944 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.950905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.950944 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.950934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmmn\" (UniqueName: \"kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.951173 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.950964 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:13.951173 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.951018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79s2n\" (UniqueName: \"kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:13.951327 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.951165 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.951327 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.951243 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:13.951449 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.951334 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.951507 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.951446 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:13.961122 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:13.961094 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmmn\" (UniqueName: \"kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:14.021777 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.021708 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8"] Sep 16 04:57:14.025506 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.025486 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.039975 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.039955 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8"] Sep 16 04:57:14.040807 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.040792 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:14.052676 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052656 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.052775 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.052775 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.052883 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052789 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79s2n\" (UniqueName: \"kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.052883 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwgk\" (UniqueName: \"kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.052983 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.052931 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.053085 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.053060 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.053135 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.053092 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.061250 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.061225 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79s2n\" (UniqueName: \"kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.134980 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.134957 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:14.153869 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.153839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.153869 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.153872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwgk\" (UniqueName: \"kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.154087 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.153901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j9x\" (UniqueName: \"kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.154087 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.153945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.154087 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.153968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.154087 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.154011 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.154309 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.154217 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.154309 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.154253 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.163173 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.163145 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwgk\" (UniqueName: \"kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.240922 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.240894 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:14.255327 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.255299 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j9x\" (UniqueName: \"kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.255439 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.255390 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.255497 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.255445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.255854 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.255819 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.255946 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.255832 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.265565 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.265532 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j9x\" (UniqueName: \"kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.318401 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.318110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7"] Sep 16 04:57:14.320785 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:57:14.320758 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cf4e23_dcf2_4a0b_878c_8f29eb68d182.slice/crio-284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f WatchSource:0}: Error finding container 284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f: Status 404 returned error can't find the container with id 284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f Sep 16 04:57:14.334359 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.334338 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:14.411867 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.411592 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn"] Sep 16 04:57:14.414305 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:57:14.414278 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9681c0d_2318_46d6_86a1_2b14f6abfad1.slice/crio-6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9 WatchSource:0}: Error finding container 6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9: Status 404 returned error can't find the container with id 6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9 Sep 16 04:57:14.519780 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.519758 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq"] Sep 16 04:57:14.521260 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:57:14.521237 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0820d7ae_92c5_463e_8dba_dffa329fb122.slice/crio-e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026 WatchSource:0}: Error finding container e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026: Status 404 returned error can't find the container with id e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026 Sep 16 04:57:14.608834 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.608801 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8"] Sep 16 04:57:14.646881 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.646854 2560 generic.go:356] "Generic (PLEG): container finished" podID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerID="a97b33965d990929de393a0a6d148cea9d84d9865fc2ddc01d5b7c5bd7a05e14" exitCode=0 Sep 16 04:57:14.646987 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.646934 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" event={"ID":"21cf4e23-dcf2-4a0b-878c-8f29eb68d182","Type":"ContainerDied","Data":"a97b33965d990929de393a0a6d148cea9d84d9865fc2ddc01d5b7c5bd7a05e14"} Sep 16 04:57:14.647046 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.646985 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" event={"ID":"21cf4e23-dcf2-4a0b-878c-8f29eb68d182","Type":"ContainerStarted","Data":"284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f"} Sep 16 04:57:14.648680 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.648657 2560 generic.go:356] "Generic (PLEG): container finished" podID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerID="488c537ed417d3709a604eaf321e8971bf2b04df39050740fa8fb9475edf6b35" exitCode=0 Sep 16 04:57:14.648762 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.648740 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" event={"ID":"0820d7ae-92c5-463e-8dba-dffa329fb122","Type":"ContainerDied","Data":"488c537ed417d3709a604eaf321e8971bf2b04df39050740fa8fb9475edf6b35"} Sep 16 04:57:14.648820 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.648786 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" event={"ID":"0820d7ae-92c5-463e-8dba-dffa329fb122","Type":"ContainerStarted","Data":"e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026"} Sep 16 04:57:14.650238 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.650218 2560 generic.go:356] "Generic (PLEG): container finished" podID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerID="de710c9b31ca241a3de8f549283a5dfd84ed9a2a71211968c94bfd018e0ca594" exitCode=0 Sep 16 04:57:14.650283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.650241 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" event={"ID":"b9681c0d-2318-46d6-86a1-2b14f6abfad1","Type":"ContainerDied","Data":"de710c9b31ca241a3de8f549283a5dfd84ed9a2a71211968c94bfd018e0ca594"} Sep 16 04:57:14.650283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:14.650279 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" event={"ID":"b9681c0d-2318-46d6-86a1-2b14f6abfad1","Type":"ContainerStarted","Data":"6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9"} Sep 16 04:57:14.678219 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:57:14.678190 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133bf9cc_2cd2_4d83_879a_dfe0841861e1.slice/crio-171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785 WatchSource:0}: Error finding container 171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785: Status 404 returned error can't find the container with id 171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785 Sep 16 04:57:15.658283 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:15.658260 2560 generic.go:356] "Generic (PLEG): container finished" podID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerID="057e3c57720a825008a33a0fc37a3734fbe3cb7e0db6f11b5996fe58f2bd462d" exitCode=0 Sep 16 04:57:15.658611 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:15.658347 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" event={"ID":"133bf9cc-2cd2-4d83-879a-dfe0841861e1","Type":"ContainerDied","Data":"057e3c57720a825008a33a0fc37a3734fbe3cb7e0db6f11b5996fe58f2bd462d"} Sep 16 04:57:15.658611 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:15.658412 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" event={"ID":"133bf9cc-2cd2-4d83-879a-dfe0841861e1","Type":"ContainerStarted","Data":"171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785"} Sep 16 04:57:16.666362 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:16.666328 2560 generic.go:356] "Generic (PLEG): container finished" podID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerID="178fe4dc44f4891df4e03afd95632e4c6d1f4233b1a6e444666ec8bae5047f59" exitCode=0 Sep 16 04:57:16.666798 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:16.666415 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" event={"ID":"b9681c0d-2318-46d6-86a1-2b14f6abfad1","Type":"ContainerDied","Data":"178fe4dc44f4891df4e03afd95632e4c6d1f4233b1a6e444666ec8bae5047f59"} Sep 16 04:57:16.668469 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:16.668447 2560 generic.go:356] "Generic (PLEG): container finished" podID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerID="eb3d526d292f3a6725c49bad45afa7bc6f581e861dc91ce030421d71c7eb5d9e" exitCode=0 Sep 16 04:57:16.668576 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:16.668501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" event={"ID":"21cf4e23-dcf2-4a0b-878c-8f29eb68d182","Type":"ContainerDied","Data":"eb3d526d292f3a6725c49bad45afa7bc6f581e861dc91ce030421d71c7eb5d9e"} Sep 16 04:57:17.675703 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.675668 2560 generic.go:356] "Generic (PLEG): container finished" podID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerID="07957c19398e9b6f552484a3fb1ba11544e85917bd7b691e460255b18d931d41" exitCode=0 Sep 16 04:57:17.676121 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.675758 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" event={"ID":"b9681c0d-2318-46d6-86a1-2b14f6abfad1","Type":"ContainerDied","Data":"07957c19398e9b6f552484a3fb1ba11544e85917bd7b691e460255b18d931d41"} Sep 16 04:57:17.677799 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.677776 2560 generic.go:356] "Generic (PLEG): container finished" podID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerID="87e20678657f0843937d6a427d7b222505b9a3c0140b7ea40d43e631488b8ac5" exitCode=0 Sep 16 04:57:17.677879 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.677859 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" event={"ID":"21cf4e23-dcf2-4a0b-878c-8f29eb68d182","Type":"ContainerDied","Data":"87e20678657f0843937d6a427d7b222505b9a3c0140b7ea40d43e631488b8ac5"} Sep 16 04:57:17.679576 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.679542 2560 generic.go:356] "Generic (PLEG): container finished" podID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerID="9ae29bb456ab874cee8ad62cd818644c488f7aef91cb53c7902e934e86241991" exitCode=0 Sep 16 04:57:17.679685 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:17.679610 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" event={"ID":"133bf9cc-2cd2-4d83-879a-dfe0841861e1","Type":"ContainerDied","Data":"9ae29bb456ab874cee8ad62cd818644c488f7aef91cb53c7902e934e86241991"} Sep 16 04:57:18.686498 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.686466 2560 generic.go:356] "Generic (PLEG): container finished" podID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerID="a4f928167600106a08aa11957c3a8c1fc12d327761d9a0f60bd5b26fc084dd02" exitCode=0 Sep 16 04:57:18.686912 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.686575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" event={"ID":"133bf9cc-2cd2-4d83-879a-dfe0841861e1","Type":"ContainerDied","Data":"a4f928167600106a08aa11957c3a8c1fc12d327761d9a0f60bd5b26fc084dd02"} Sep 16 04:57:18.831411 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.831388 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:18.834124 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.834106 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:18.895818 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895793 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtmmn\" (UniqueName: \"kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn\") pod \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " Sep 16 04:57:18.895939 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895832 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util\") pod \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " Sep 16 04:57:18.895939 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895853 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle\") pod \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\" (UID: \"21cf4e23-dcf2-4a0b-878c-8f29eb68d182\") " Sep 16 04:57:18.895939 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895873 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util\") pod \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " Sep 16 04:57:18.895939 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895923 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79s2n\" (UniqueName: \"kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n\") pod \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " Sep 16 04:57:18.896146 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.895948 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle\") pod \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\" (UID: \"b9681c0d-2318-46d6-86a1-2b14f6abfad1\") " Sep 16 04:57:18.896393 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.896365 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle" (OuterVolumeSpecName: "bundle") pod "21cf4e23-dcf2-4a0b-878c-8f29eb68d182" (UID: "21cf4e23-dcf2-4a0b-878c-8f29eb68d182"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:18.896796 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.896752 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle" (OuterVolumeSpecName: "bundle") pod "b9681c0d-2318-46d6-86a1-2b14f6abfad1" (UID: "b9681c0d-2318-46d6-86a1-2b14f6abfad1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:18.897940 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.897912 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn" (OuterVolumeSpecName: "kube-api-access-vtmmn") pod "21cf4e23-dcf2-4a0b-878c-8f29eb68d182" (UID: "21cf4e23-dcf2-4a0b-878c-8f29eb68d182"). InnerVolumeSpecName "kube-api-access-vtmmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:57:18.898041 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.898019 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n" (OuterVolumeSpecName: "kube-api-access-79s2n") pod "b9681c0d-2318-46d6-86a1-2b14f6abfad1" (UID: "b9681c0d-2318-46d6-86a1-2b14f6abfad1"). InnerVolumeSpecName "kube-api-access-79s2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:57:18.901510 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.901489 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util" (OuterVolumeSpecName: "util") pod "21cf4e23-dcf2-4a0b-878c-8f29eb68d182" (UID: "21cf4e23-dcf2-4a0b-878c-8f29eb68d182"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:18.904166 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.904143 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util" (OuterVolumeSpecName: "util") pod "b9681c0d-2318-46d6-86a1-2b14f6abfad1" (UID: "b9681c0d-2318-46d6-86a1-2b14f6abfad1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996693 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996713 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtmmn\" (UniqueName: \"kubernetes.io/projected/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-kube-api-access-vtmmn\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996723 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996732 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21cf4e23-dcf2-4a0b-878c-8f29eb68d182-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996740 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9681c0d-2318-46d6-86a1-2b14f6abfad1-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:18.996744 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:18.996748 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79s2n\" (UniqueName: \"kubernetes.io/projected/b9681c0d-2318-46d6-86a1-2b14f6abfad1-kube-api-access-79s2n\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:19.693845 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.693819 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" Sep 16 04:57:19.694268 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.693855 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fdzx7" event={"ID":"21cf4e23-dcf2-4a0b-878c-8f29eb68d182","Type":"ContainerDied","Data":"284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f"} Sep 16 04:57:19.694268 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.693906 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284ecc97a6330f307022ed9768fd7aeabe46f7d0c95d7af26564269a45c0541f" Sep 16 04:57:19.695911 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.695876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" event={"ID":"b9681c0d-2318-46d6-86a1-2b14f6abfad1","Type":"ContainerDied","Data":"6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9"} Sep 16 04:57:19.695911 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.695927 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b2a6c70cfca569d7be9e8405d047af941736497a1ea76834df9746d1bd7acf9" Sep 16 04:57:19.696131 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.696116 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7xzmn" Sep 16 04:57:19.807359 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.807324 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:19.903769 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.903738 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util\") pod \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " Sep 16 04:57:19.903919 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.903806 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64j9x\" (UniqueName: \"kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x\") pod \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " Sep 16 04:57:19.903919 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.903824 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle\") pod \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\" (UID: \"133bf9cc-2cd2-4d83-879a-dfe0841861e1\") " Sep 16 04:57:19.904373 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.904347 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle" (OuterVolumeSpecName: "bundle") pod "133bf9cc-2cd2-4d83-879a-dfe0841861e1" (UID: "133bf9cc-2cd2-4d83-879a-dfe0841861e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:19.905866 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.905846 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x" (OuterVolumeSpecName: "kube-api-access-64j9x") pod "133bf9cc-2cd2-4d83-879a-dfe0841861e1" (UID: "133bf9cc-2cd2-4d83-879a-dfe0841861e1"). InnerVolumeSpecName "kube-api-access-64j9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:57:19.909294 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:19.909256 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util" (OuterVolumeSpecName: "util") pod "133bf9cc-2cd2-4d83-879a-dfe0841861e1" (UID: "133bf9cc-2cd2-4d83-879a-dfe0841861e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:20.004420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.004345 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:20.004420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.004376 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64j9x\" (UniqueName: \"kubernetes.io/projected/133bf9cc-2cd2-4d83-879a-dfe0841861e1-kube-api-access-64j9x\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:20.004420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.004390 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/133bf9cc-2cd2-4d83-879a-dfe0841861e1-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:20.702888 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.702852 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" event={"ID":"133bf9cc-2cd2-4d83-879a-dfe0841861e1","Type":"ContainerDied","Data":"171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785"} Sep 16 04:57:20.703369 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.702904 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="171463b672b292db63e72976b7cc27f5bfe7b75b9d29026658234d947ceaf785" Sep 16 04:57:20.703369 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:20.702861 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw6rn8" Sep 16 04:57:23.719863 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:23.719830 2560 generic.go:356] "Generic (PLEG): container finished" podID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerID="3c0bebef3283d23299b6685ec1c929b08573f6cfde25d63f10930cb86bd56ff9" exitCode=0 Sep 16 04:57:23.720337 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:23.719873 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" event={"ID":"0820d7ae-92c5-463e-8dba-dffa329fb122","Type":"ContainerDied","Data":"3c0bebef3283d23299b6685ec1c929b08573f6cfde25d63f10930cb86bd56ff9"} Sep 16 04:57:24.726900 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:24.726862 2560 generic.go:356] "Generic (PLEG): container finished" podID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerID="da5a88c17ae86408ef3ece0e192a481b681386987cf549fed042d2444296992e" exitCode=0 Sep 16 04:57:24.727296 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:24.726947 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" event={"ID":"0820d7ae-92c5-463e-8dba-dffa329fb122","Type":"ContainerDied","Data":"da5a88c17ae86408ef3ece0e192a481b681386987cf549fed042d2444296992e"} Sep 16 04:57:25.858750 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.858725 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:25.951864 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.951832 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util\") pod \"0820d7ae-92c5-463e-8dba-dffa329fb122\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " Sep 16 04:57:25.952029 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.951890 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hwgk\" (UniqueName: \"kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk\") pod \"0820d7ae-92c5-463e-8dba-dffa329fb122\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " Sep 16 04:57:25.952029 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.951935 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle\") pod \"0820d7ae-92c5-463e-8dba-dffa329fb122\" (UID: \"0820d7ae-92c5-463e-8dba-dffa329fb122\") " Sep 16 04:57:25.952518 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.952487 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle" (OuterVolumeSpecName: "bundle") pod "0820d7ae-92c5-463e-8dba-dffa329fb122" (UID: "0820d7ae-92c5-463e-8dba-dffa329fb122"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:25.954018 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.953996 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk" (OuterVolumeSpecName: "kube-api-access-9hwgk") pod "0820d7ae-92c5-463e-8dba-dffa329fb122" (UID: "0820d7ae-92c5-463e-8dba-dffa329fb122"). InnerVolumeSpecName "kube-api-access-9hwgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:57:25.956297 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:25.956266 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util" (OuterVolumeSpecName: "util") pod "0820d7ae-92c5-463e-8dba-dffa329fb122" (UID: "0820d7ae-92c5-463e-8dba-dffa329fb122"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 04:57:26.053259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.053176 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-util\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:26.053259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.053230 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwgk\" (UniqueName: \"kubernetes.io/projected/0820d7ae-92c5-463e-8dba-dffa329fb122-kube-api-access-9hwgk\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:26.053259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.053242 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0820d7ae-92c5-463e-8dba-dffa329fb122-bundle\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:57:26.738435 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.738405 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" Sep 16 04:57:26.738435 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.738425 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7t2jzq" event={"ID":"0820d7ae-92c5-463e-8dba-dffa329fb122","Type":"ContainerDied","Data":"e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026"} Sep 16 04:57:26.738673 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:26.738471 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4de2f4e611489967d63cea79531e56146012327ea9671a4d366083201077026" Sep 16 04:57:52.068616 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068524 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq"] Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068943 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068957 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068966 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068971 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068979 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="extract" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068985 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="extract" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.068995 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="pull" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069001 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="pull" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069010 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069015 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069025 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069030 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="util" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069035 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="pull" Sep 16 04:57:52.069034 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069040 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="pull" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069049 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="pull" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069054 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="pull" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069061 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069066 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069071 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069075 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069081 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="pull" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069086 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="pull" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069091 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069096 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069148 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9681c0d-2318-46d6-86a1-2b14f6abfad1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069156 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0820d7ae-92c5-463e-8dba-dffa329fb122" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069163 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="133bf9cc-2cd2-4d83-879a-dfe0841861e1" containerName="extract" Sep 16 04:57:52.069541 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.069169 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="21cf4e23-dcf2-4a0b-878c-8f29eb68d182" containerName="extract" Sep 16 04:57:52.072117 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.072100 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.074572 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.074532 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 04:57:52.074698 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.074670 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 04:57:52.075708 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.075688 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d2fln\"" Sep 16 04:57:52.075708 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.075705 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 04:57:52.075847 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.075718 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 04:57:52.079930 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.079910 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq"] Sep 16 04:57:52.178246 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.178220 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.178357 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.178260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tck\" (UniqueName: \"kubernetes.io/projected/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-kube-api-access-w2tck\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.178402 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.178348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.279602 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.279573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tck\" (UniqueName: \"kubernetes.io/projected/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-kube-api-access-w2tck\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.279740 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.279647 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.279740 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.279684 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.279831 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:57:52.279763 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 16 04:57:52.279831 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:57:52.279812 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert podName:0789c1c8-920e-4ef2-acbf-7168e7cf3b26 nodeName:}" failed. No retries permitted until 2026-09-16 04:57:52.779798104 +0000 UTC m=+538.293349616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-blzpq" (UID: "0789c1c8-920e-4ef2-acbf-7168e7cf3b26") : secret "plugin-serving-cert" not found Sep 16 04:57:52.280254 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.280235 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.288254 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.288233 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tck\" (UniqueName: \"kubernetes.io/projected/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-kube-api-access-w2tck\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.783706 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.783674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.786052 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.786032 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0789c1c8-920e-4ef2-acbf-7168e7cf3b26-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-blzpq\" (UID: \"0789c1c8-920e-4ef2-acbf-7168e7cf3b26\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:52.982420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:52.982389 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" Sep 16 04:57:53.253979 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:53.253956 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq"] Sep 16 04:57:53.256113 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:57:53.256085 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0789c1c8_920e_4ef2_acbf_7168e7cf3b26.slice/crio-1fb1d930ddd4f9935ac62ca32bfb1c741ed0c8cbb3f99a8a0e8c167cfc7281b3 WatchSource:0}: Error finding container 1fb1d930ddd4f9935ac62ca32bfb1c741ed0c8cbb3f99a8a0e8c167cfc7281b3: Status 404 returned error can't find the container with id 1fb1d930ddd4f9935ac62ca32bfb1c741ed0c8cbb3f99a8a0e8c167cfc7281b3 Sep 16 04:57:53.882475 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:53.882440 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" event={"ID":"0789c1c8-920e-4ef2-acbf-7168e7cf3b26","Type":"ContainerStarted","Data":"1fb1d930ddd4f9935ac62ca32bfb1c741ed0c8cbb3f99a8a0e8c167cfc7281b3"} Sep 16 04:57:57.908261 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:57.908234 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" event={"ID":"0789c1c8-920e-4ef2-acbf-7168e7cf3b26","Type":"ContainerStarted","Data":"d08f7b401376b7f489735f09ca513753d160562a323445d3c15bf21405646fc9"} Sep 16 04:57:57.924651 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:57:57.924604 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-blzpq" podStartSLOduration=1.359860672 podStartE2EDuration="5.924587938s" totalImagesPullingTime="4.564727266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:57:57.923671833 +0000 UTC m=+543.437223368" watchObservedRunningTime="2026-09-16 04:57:57.924587938 +0000 UTC m=+543.438139469" Sep 16 04:58:48.971969 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:48.971933 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:48.977013 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:48.976993 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:48.979849 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:48.979828 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pdpv5\"" Sep 16 04:58:48.982068 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:48.982037 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:49.075027 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.075002 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lptc6\" (UniqueName: \"kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6\") pod \"authorino-7bcfd8f9cd-9cbrb\" (UID: \"1ccaf8e1-81fa-4297-b180-17e7a2298feb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:49.100001 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.099976 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:58:49.103194 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.103180 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:58:49.111360 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.111329 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:58:49.175793 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.175766 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lptc6\" (UniqueName: \"kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6\") pod \"authorino-7bcfd8f9cd-9cbrb\" (UID: \"1ccaf8e1-81fa-4297-b180-17e7a2298feb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:49.183730 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.183710 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lptc6\" (UniqueName: \"kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6\") pod \"authorino-7bcfd8f9cd-9cbrb\" (UID: \"1ccaf8e1-81fa-4297-b180-17e7a2298feb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:49.276659 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.276593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w54s\" (UniqueName: \"kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s\") pod \"authorino-59c974bd48-5thqj\" (UID: \"e4d03e31-83e1-49df-acdf-a0e0f298276b\") " pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:58:49.287432 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.287412 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:49.377859 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.377822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w54s\" (UniqueName: \"kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s\") pod \"authorino-59c974bd48-5thqj\" (UID: \"e4d03e31-83e1-49df-acdf-a0e0f298276b\") " pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:58:49.386933 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.386907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w54s\" (UniqueName: \"kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s\") pod \"authorino-59c974bd48-5thqj\" (UID: \"e4d03e31-83e1-49df-acdf-a0e0f298276b\") " pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:58:49.412770 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.412737 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:58:49.557993 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.557957 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:49.559669 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:58:49.559639 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ccaf8e1_81fa_4297_b180_17e7a2298feb.slice/crio-8c5b8e8a800e6ac89e0fd1d97480a588b30003acc96f936606856b07e8b50f82 WatchSource:0}: Error finding container 8c5b8e8a800e6ac89e0fd1d97480a588b30003acc96f936606856b07e8b50f82: Status 404 returned error can't find the container with id 8c5b8e8a800e6ac89e0fd1d97480a588b30003acc96f936606856b07e8b50f82 Sep 16 04:58:49.683440 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:49.683416 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:58:49.684990 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:58:49.684966 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d03e31_83e1_49df_acdf_a0e0f298276b.slice/crio-26354bcf898b9f983f1819a94a2fe4e2462311587039bba7c73511c3d52e7d3b WatchSource:0}: Error finding container 26354bcf898b9f983f1819a94a2fe4e2462311587039bba7c73511c3d52e7d3b: Status 404 returned error can't find the container with id 26354bcf898b9f983f1819a94a2fe4e2462311587039bba7c73511c3d52e7d3b Sep 16 04:58:50.191904 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:50.191844 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5thqj" event={"ID":"e4d03e31-83e1-49df-acdf-a0e0f298276b","Type":"ContainerStarted","Data":"26354bcf898b9f983f1819a94a2fe4e2462311587039bba7c73511c3d52e7d3b"} Sep 16 04:58:50.194796 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:50.194753 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" event={"ID":"1ccaf8e1-81fa-4297-b180-17e7a2298feb","Type":"ContainerStarted","Data":"8c5b8e8a800e6ac89e0fd1d97480a588b30003acc96f936606856b07e8b50f82"} Sep 16 04:58:53.213056 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:53.213017 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5thqj" event={"ID":"e4d03e31-83e1-49df-acdf-a0e0f298276b","Type":"ContainerStarted","Data":"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b"} Sep 16 04:58:53.214724 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:53.214699 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" event={"ID":"1ccaf8e1-81fa-4297-b180-17e7a2298feb","Type":"ContainerStarted","Data":"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9"} Sep 16 04:58:53.230259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:53.230216 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-5thqj" podStartSLOduration=1.6507840520000001 podStartE2EDuration="4.230202326s" totalImagesPullingTime="2.579418274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:53.228151573 +0000 UTC m=+598.741703108" watchObservedRunningTime="2026-09-16 04:58:53.230202326 +0000 UTC m=+598.743753859" Sep 16 04:58:53.243256 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:53.243214 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" podStartSLOduration=2.5275337369999997 podStartE2EDuration="5.243201959s" totalImagesPullingTime="2.715668222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:58:53.241172987 +0000 UTC m=+598.754724524" watchObservedRunningTime="2026-09-16 04:58:53.243201959 +0000 UTC m=+598.756753493" Sep 16 04:58:53.266587 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:53.266564 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:54.947497 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:54.947470 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:58:54.950427 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:54.950405 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:58:54.956209 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:54.956184 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 04:58:54.959462 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:54.959445 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 04:58:55.225411 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:55.225332 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" podUID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" containerName="authorino" containerID="cri-o://e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9" gracePeriod=30 Sep 16 04:58:55.465400 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:55.465374 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:55.536086 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:55.536012 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lptc6\" (UniqueName: \"kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6\") pod \"1ccaf8e1-81fa-4297-b180-17e7a2298feb\" (UID: \"1ccaf8e1-81fa-4297-b180-17e7a2298feb\") " Sep 16 04:58:55.538068 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:55.538045 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6" (OuterVolumeSpecName: "kube-api-access-lptc6") pod "1ccaf8e1-81fa-4297-b180-17e7a2298feb" (UID: "1ccaf8e1-81fa-4297-b180-17e7a2298feb"). InnerVolumeSpecName "kube-api-access-lptc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:58:55.636674 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:55.636647 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lptc6\" (UniqueName: \"kubernetes.io/projected/1ccaf8e1-81fa-4297-b180-17e7a2298feb-kube-api-access-lptc6\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:58:56.231513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.231469 2560 generic.go:356] "Generic (PLEG): container finished" podID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" containerID="e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9" exitCode=0 Sep 16 04:58:56.231513 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.231511 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" Sep 16 04:58:56.231993 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.231564 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" event={"ID":"1ccaf8e1-81fa-4297-b180-17e7a2298feb","Type":"ContainerDied","Data":"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9"} Sep 16 04:58:56.231993 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.231612 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9cbrb" event={"ID":"1ccaf8e1-81fa-4297-b180-17e7a2298feb","Type":"ContainerDied","Data":"8c5b8e8a800e6ac89e0fd1d97480a588b30003acc96f936606856b07e8b50f82"} Sep 16 04:58:56.231993 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.231628 2560 scope.go:117] "RemoveContainer" containerID="e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9" Sep 16 04:58:56.241629 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.241614 2560 scope.go:117] "RemoveContainer" containerID="e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9" Sep 16 04:58:56.241892 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:58:56.241860 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9\": container with ID starting with e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9 not found: ID does not exist" containerID="e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9" Sep 16 04:58:56.241974 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.241905 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9"} err="failed to get container status \"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9\": rpc error: code = NotFound desc = could not find container \"e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9\": container with ID starting with e91ca1b919adc9eca7537bd1ce75cfe2d7062b845fcca1d6cf17e0c098dcb1a9 not found: ID does not exist" Sep 16 04:58:56.253252 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.253228 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:56.257811 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:56.257791 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9cbrb"] Sep 16 04:58:57.120402 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:58:57.120370 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" path="/var/lib/kubelet/pods/1ccaf8e1-81fa-4297-b180-17e7a2298feb/volumes" Sep 16 04:59:18.142255 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.142168 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:18.142742 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.142528 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" containerName="authorino" Sep 16 04:59:18.142742 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.142539 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" containerName="authorino" Sep 16 04:59:18.142742 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.142612 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ccaf8e1-81fa-4297-b180-17e7a2298feb" containerName="authorino" Sep 16 04:59:18.151355 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.151321 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:18.151912 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.151883 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:18.225223 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.225187 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbknq\" (UniqueName: \"kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq\") pod \"authorino-557f876f69-5zmd7\" (UID: \"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17\") " pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:18.326657 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.326615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbknq\" (UniqueName: \"kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq\") pod \"authorino-557f876f69-5zmd7\" (UID: \"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17\") " pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:18.337076 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.337047 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbknq\" (UniqueName: \"kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq\") pod \"authorino-557f876f69-5zmd7\" (UID: \"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17\") " pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:18.368327 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.368294 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:18.368608 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.368593 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:18.397504 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.397437 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-645c84ff59-4kd29"] Sep 16 04:59:18.402593 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.402568 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:18.410023 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.409993 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-645c84ff59-4kd29"] Sep 16 04:59:18.478971 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.478935 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-645c84ff59-4kd29"] Sep 16 04:59:18.479255 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:59:18.479234 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cn47l], unattached volumes=[], failed to process volumes=[kube-api-access-cn47l]: context canceled" pod="kuadrant-system/authorino-645c84ff59-4kd29" podUID="ca9d3c93-7709-49a2-989b-236875fbfde7" Sep 16 04:59:18.508376 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.508329 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 04:59:18.520030 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.519995 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 04:59:18.520200 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.520109 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.522822 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.522799 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 04:59:18.529398 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.529364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn47l\" (UniqueName: \"kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l\") pod \"authorino-645c84ff59-4kd29\" (UID: \"ca9d3c93-7709-49a2-989b-236875fbfde7\") " pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:18.630104 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.630074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8nr\" (UniqueName: \"kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.630259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.630139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn47l\" (UniqueName: \"kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l\") pod \"authorino-645c84ff59-4kd29\" (UID: \"ca9d3c93-7709-49a2-989b-236875fbfde7\") " pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:18.630259 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.630160 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.637817 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.637793 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn47l\" (UniqueName: \"kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l\") pod \"authorino-645c84ff59-4kd29\" (UID: \"ca9d3c93-7709-49a2-989b-236875fbfde7\") " pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:18.642935 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.642920 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:18.645808 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:59:18.645781 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb1d990_99e5_48ef_a4b9_c1a8a8d96e17.slice/crio-96a108537e233213889427d27a6cd6983a1d74aff3adcf53ca1ad6ab9cd5214c WatchSource:0}: Error finding container 96a108537e233213889427d27a6cd6983a1d74aff3adcf53ca1ad6ab9cd5214c: Status 404 returned error can't find the container with id 96a108537e233213889427d27a6cd6983a1d74aff3adcf53ca1ad6ab9cd5214c Sep 16 04:59:18.731564 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.731526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8nr\" (UniqueName: \"kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.731732 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.731615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.733968 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.733947 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.739201 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.739182 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8nr\" (UniqueName: \"kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr\") pod \"authorino-77985d8bf7-wr8jg\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:18.829815 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:18.829782 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 04:59:19.103274 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.103247 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 04:59:19.106683 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:59:19.106658 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf67c98_186e_4b5c_80d7_eac7fda3974e.slice/crio-1c8cb8e9bdf9d0489f9b12a8cfb45b35f04a01a0b4ef41b5f136011036c00601 WatchSource:0}: Error finding container 1c8cb8e9bdf9d0489f9b12a8cfb45b35f04a01a0b4ef41b5f136011036c00601: Status 404 returned error can't find the container with id 1c8cb8e9bdf9d0489f9b12a8cfb45b35f04a01a0b4ef41b5f136011036c00601 Sep 16 04:59:19.356672 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.356579 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-5zmd7" event={"ID":"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17","Type":"ContainerStarted","Data":"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55"} Sep 16 04:59:19.356672 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.356580 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-5zmd7" podUID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" containerName="authorino" containerID="cri-o://c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55" gracePeriod=30 Sep 16 04:59:19.356672 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.356627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-5zmd7" event={"ID":"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17","Type":"ContainerStarted","Data":"96a108537e233213889427d27a6cd6983a1d74aff3adcf53ca1ad6ab9cd5214c"} Sep 16 04:59:19.358002 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.357974 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" event={"ID":"2cf67c98-186e-4b5c-80d7-eac7fda3974e","Type":"ContainerStarted","Data":"1c8cb8e9bdf9d0489f9b12a8cfb45b35f04a01a0b4ef41b5f136011036c00601"} Sep 16 04:59:19.358002 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.357993 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:19.365234 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.365214 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:19.373046 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.373009 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-5zmd7" podStartSLOduration=0.911399867 podStartE2EDuration="1.37299545s" totalImagesPullingTime="461.595583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:59:19.370994682 +0000 UTC m=+624.884546240" watchObservedRunningTime="2026-09-16 04:59:19.37299545 +0000 UTC m=+624.886546984" Sep 16 04:59:19.437169 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.437143 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn47l\" (UniqueName: \"kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l\") pod \"ca9d3c93-7709-49a2-989b-236875fbfde7\" (UID: \"ca9d3c93-7709-49a2-989b-236875fbfde7\") " Sep 16 04:59:19.439172 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.439139 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l" (OuterVolumeSpecName: "kube-api-access-cn47l") pod "ca9d3c93-7709-49a2-989b-236875fbfde7" (UID: "ca9d3c93-7709-49a2-989b-236875fbfde7"). InnerVolumeSpecName "kube-api-access-cn47l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:59:19.538041 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.538014 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn47l\" (UniqueName: \"kubernetes.io/projected/ca9d3c93-7709-49a2-989b-236875fbfde7-kube-api-access-cn47l\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:59:19.601714 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.601382 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:19.638502 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.638361 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbknq\" (UniqueName: \"kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq\") pod \"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17\" (UID: \"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17\") " Sep 16 04:59:19.640560 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.640515 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq" (OuterVolumeSpecName: "kube-api-access-vbknq") pod "3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" (UID: "3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17"). InnerVolumeSpecName "kube-api-access-vbknq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:59:19.739896 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:19.739854 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbknq\" (UniqueName: \"kubernetes.io/projected/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17-kube-api-access-vbknq\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:59:20.365771 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.365731 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" event={"ID":"2cf67c98-186e-4b5c-80d7-eac7fda3974e","Type":"ContainerStarted","Data":"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a"} Sep 16 04:59:20.366936 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.366915 2560 generic.go:356] "Generic (PLEG): container finished" podID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" containerID="c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55" exitCode=0 Sep 16 04:59:20.367006 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.366950 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-5zmd7" Sep 16 04:59:20.367006 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.366986 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-5zmd7" event={"ID":"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17","Type":"ContainerDied","Data":"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55"} Sep 16 04:59:20.367073 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.367014 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-5zmd7" event={"ID":"3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17","Type":"ContainerDied","Data":"96a108537e233213889427d27a6cd6983a1d74aff3adcf53ca1ad6ab9cd5214c"} Sep 16 04:59:20.367073 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.367030 2560 scope.go:117] "RemoveContainer" containerID="c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55" Sep 16 04:59:20.367197 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.367152 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645c84ff59-4kd29" Sep 16 04:59:20.376950 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.376935 2560 scope.go:117] "RemoveContainer" containerID="c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55" Sep 16 04:59:20.377206 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:59:20.377188 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55\": container with ID starting with c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55 not found: ID does not exist" containerID="c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55" Sep 16 04:59:20.377278 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.377212 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55"} err="failed to get container status \"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55\": rpc error: code = NotFound desc = could not find container \"c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55\": container with ID starting with c0029625d894a1a8a939fcdff5a9bfc52e58a544efc3c339b677be8055585d55 not found: ID does not exist" Sep 16 04:59:20.384801 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.384764 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" podStartSLOduration=2.010862522 podStartE2EDuration="2.384751971s" totalImagesPullingTime="373.889449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:59:20.381540008 +0000 UTC m=+625.895091556" watchObservedRunningTime="2026-09-16 04:59:20.384751971 +0000 UTC m=+625.898303505" Sep 16 04:59:20.405759 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.405729 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:59:20.405917 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.405895 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-5thqj" podUID="e4d03e31-83e1-49df-acdf-a0e0f298276b" containerName="authorino" containerID="cri-o://d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b" gracePeriod=30 Sep 16 04:59:20.408120 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.408101 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-645c84ff59-4kd29"] Sep 16 04:59:20.413989 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.413966 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-645c84ff59-4kd29"] Sep 16 04:59:20.426722 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.426696 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:20.432644 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.432621 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-5zmd7"] Sep 16 04:59:20.655578 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.655538 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:59:20.748893 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.748850 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w54s\" (UniqueName: \"kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s\") pod \"e4d03e31-83e1-49df-acdf-a0e0f298276b\" (UID: \"e4d03e31-83e1-49df-acdf-a0e0f298276b\") " Sep 16 04:59:20.750950 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.750925 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s" (OuterVolumeSpecName: "kube-api-access-8w54s") pod "e4d03e31-83e1-49df-acdf-a0e0f298276b" (UID: "e4d03e31-83e1-49df-acdf-a0e0f298276b"). InnerVolumeSpecName "kube-api-access-8w54s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 04:59:20.850420 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:20.850382 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w54s\" (UniqueName: \"kubernetes.io/projected/e4d03e31-83e1-49df-acdf-a0e0f298276b-kube-api-access-8w54s\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 04:59:21.122162 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.122126 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" path="/var/lib/kubelet/pods/3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17/volumes" Sep 16 04:59:21.122519 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.122502 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9d3c93-7709-49a2-989b-236875fbfde7" path="/var/lib/kubelet/pods/ca9d3c93-7709-49a2-989b-236875fbfde7/volumes" Sep 16 04:59:21.377630 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.377519 2560 generic.go:356] "Generic (PLEG): container finished" podID="e4d03e31-83e1-49df-acdf-a0e0f298276b" containerID="d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b" exitCode=0 Sep 16 04:59:21.377630 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.377591 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-5thqj" Sep 16 04:59:21.377630 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.377591 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5thqj" event={"ID":"e4d03e31-83e1-49df-acdf-a0e0f298276b","Type":"ContainerDied","Data":"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b"} Sep 16 04:59:21.378177 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.377652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-5thqj" event={"ID":"e4d03e31-83e1-49df-acdf-a0e0f298276b","Type":"ContainerDied","Data":"26354bcf898b9f983f1819a94a2fe4e2462311587039bba7c73511c3d52e7d3b"} Sep 16 04:59:21.378177 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.377674 2560 scope.go:117] "RemoveContainer" containerID="d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b" Sep 16 04:59:21.388488 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.388465 2560 scope.go:117] "RemoveContainer" containerID="d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b" Sep 16 04:59:21.388865 ip-10-0-142-144 kubenswrapper[2560]: E0916 04:59:21.388845 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b\": container with ID starting with d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b not found: ID does not exist" containerID="d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b" Sep 16 04:59:21.388937 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.388874 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b"} err="failed to get container status \"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b\": rpc error: code = NotFound desc = could not find container \"d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b\": container with ID starting with d78d27911094748a1e44853722b218fac721d291232faa9f448089600dc0578b not found: ID does not exist" Sep 16 04:59:21.396058 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.396031 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:59:21.398474 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:21.398450 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-5thqj"] Sep 16 04:59:23.121141 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:23.121096 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d03e31-83e1-49df-acdf-a0e0f298276b" path="/var/lib/kubelet/pods/e4d03e31-83e1-49df-acdf-a0e0f298276b/volumes" Sep 16 04:59:25.278117 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278088 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278411 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d03e31-83e1-49df-acdf-a0e0f298276b" containerName="authorino" Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278421 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d03e31-83e1-49df-acdf-a0e0f298276b" containerName="authorino" Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278432 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" containerName="authorino" Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278438 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" containerName="authorino" Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278523 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fb1d990-99e5-48ef-a4b9-c1a8a8d96e17" containerName="authorino" Sep 16 04:59:25.278592 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.278531 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d03e31-83e1-49df-acdf-a0e0f298276b" containerName="authorino" Sep 16 04:59:25.283026 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.283009 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.285473 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.285452 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 04:59:25.286728 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.286712 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 04:59:25.286846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.286749 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mx82w\"" Sep 16 04:59:25.286846 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.286781 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 04:59:25.290909 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.290464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 04:59:25.393099 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.393073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52cw5\" (UniqueName: \"kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.393251 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.393115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.393251 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.393218 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.494655 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.494626 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.494791 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.494681 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52cw5\" (UniqueName: \"kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.494791 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.494706 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.496954 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.496930 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.497094 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.497072 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.502155 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.502134 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52cw5\" (UniqueName: \"kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5\") pod \"maas-controller-59fbdd984-2gfk2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.594893 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.594860 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:25.868450 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:25.868427 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 04:59:25.870303 ip-10-0-142-144 kubenswrapper[2560]: W0916 04:59:25.870275 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c80e8f_a4f6_4292_89b7_6a2cb16029f2.slice/crio-30728fb58b996cf07ca41d8c2b89272dcb0787c7ae95b8fe3547232769193460 WatchSource:0}: Error finding container 30728fb58b996cf07ca41d8c2b89272dcb0787c7ae95b8fe3547232769193460: Status 404 returned error can't find the container with id 30728fb58b996cf07ca41d8c2b89272dcb0787c7ae95b8fe3547232769193460 Sep 16 04:59:26.407876 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:26.407833 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-2gfk2" event={"ID":"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2","Type":"ContainerStarted","Data":"30728fb58b996cf07ca41d8c2b89272dcb0787c7ae95b8fe3547232769193460"} Sep 16 04:59:28.421326 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:28.421288 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-2gfk2" event={"ID":"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2","Type":"ContainerStarted","Data":"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f"} Sep 16 04:59:28.421768 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:28.421415 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 04:59:28.468437 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:28.468346 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59fbdd984-2gfk2" podStartSLOduration=1.126980169 podStartE2EDuration="3.468331144s" totalImagesPullingTime="2.341350975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 04:59:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 04:59:28.466508296 +0000 UTC m=+633.980059828" watchObservedRunningTime="2026-09-16 04:59:28.468331144 +0000 UTC m=+633.981882680" Sep 16 04:59:39.428754 ip-10-0-142-144 kubenswrapper[2560]: I0916 04:59:39.428720 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 05:00:01.697151 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.697112 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:00:01.706389 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.706361 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.709172 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.709151 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 05:00:01.709172 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.709170 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 05:00:01.709361 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.709160 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 05:00:01.709361 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.709194 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 05:00:01.710595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.710572 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qdjvt\"" Sep 16 05:00:01.714072 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.713576 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:00:01.802881 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.802850 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sljf5\" (UniqueName: \"kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.803039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.802904 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.803039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.802971 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.803039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.802989 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.903676 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.903648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sljf5\" (UniqueName: \"kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.903847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.903687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.903847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.903742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.903847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.903760 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.903987 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:00:01.903860 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Sep 16 05:00:01.904090 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:00:01.904070 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls podName:4454818e-aca4-43c6-ba71-0657a50b749b nodeName:}" failed. No retries permitted until 2026-09-16 05:00:02.403947952 +0000 UTC m=+667.917499479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls") pod "maas-api-86c4f95b8f-vq662" (UID: "4454818e-aca4-43c6-ba71-0657a50b749b") : secret "maas-api-serving-cert" not found Sep 16 05:00:01.904475 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.904455 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.906596 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.906578 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:01.917450 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:01.917426 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sljf5\" (UniqueName: \"kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:02.409530 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:02.409494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:02.412014 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:02.411990 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") pod \"maas-api-86c4f95b8f-vq662\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:02.618976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:02.618938 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:02.900209 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:02.900178 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:00:02.901686 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:00:02.901647 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4454818e_aca4_43c6_ba71_0657a50b749b.slice/crio-f62ced7d42c277c7f231523d187afd3c1573bed55f5694cc0fba0cbe9996e918 WatchSource:0}: Error finding container f62ced7d42c277c7f231523d187afd3c1573bed55f5694cc0fba0cbe9996e918: Status 404 returned error can't find the container with id f62ced7d42c277c7f231523d187afd3c1573bed55f5694cc0fba0cbe9996e918 Sep 16 05:00:02.903058 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:02.903039 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 05:00:03.616391 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:03.616355 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" event={"ID":"4454818e-aca4-43c6-ba71-0657a50b749b","Type":"ContainerStarted","Data":"f62ced7d42c277c7f231523d187afd3c1573bed55f5694cc0fba0cbe9996e918"} Sep 16 05:00:06.635250 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:06.635216 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" event={"ID":"4454818e-aca4-43c6-ba71-0657a50b749b","Type":"ContainerStarted","Data":"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a"} Sep 16 05:00:06.635690 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:06.635357 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:06.656024 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:06.655980 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" podStartSLOduration=2.728544662 podStartE2EDuration="5.655966421s" totalImagesPullingTime="2.927421759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:00:06.653601214 +0000 UTC m=+672.167152749" watchObservedRunningTime="2026-09-16 05:00:06.655966421 +0000 UTC m=+672.169517955" Sep 16 05:00:12.652052 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:12.652021 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:00:29.853954 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.853915 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc"] Sep 16 05:00:29.857520 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.857503 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.860379 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.860359 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 05:00:29.861375 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.861354 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sl6zs\"" Sep 16 05:00:29.861484 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.861374 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 16 05:00:29.861484 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.861393 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 05:00:29.868430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.868406 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc"] Sep 16 05:00:29.958685 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958655 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcf1429-ec8b-4377-aa14-438236a2df38-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.958844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958711 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl92\" (UniqueName: \"kubernetes.io/projected/abcf1429-ec8b-4377-aa14-438236a2df38-kube-api-access-zcl92\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.958844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958761 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.958844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958796 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.958844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:29.958844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:29.958839 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060020 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.059977 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcf1429-ec8b-4377-aa14-438236a2df38-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060396 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060228 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl92\" (UniqueName: \"kubernetes.io/projected/abcf1429-ec8b-4377-aa14-438236a2df38-kube-api-access-zcl92\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060489 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060462 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060573 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.060573 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.060534 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.062404 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.062375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcf1429-ec8b-4377-aa14-438236a2df38-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.062811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.062791 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcf1429-ec8b-4377-aa14-438236a2df38-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.068955 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.068934 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl92\" (UniqueName: \"kubernetes.io/projected/abcf1429-ec8b-4377-aa14-438236a2df38-kube-api-access-zcl92\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc\" (UID: \"abcf1429-ec8b-4377-aa14-438236a2df38\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.168629 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.168521 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:30.444843 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.444777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc"] Sep 16 05:00:30.446056 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:00:30.446003 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcf1429_ec8b_4377_aa14_438236a2df38.slice/crio-07faf6fba47a0d054acd19a24a7b84a03664c9a82aeb4af4ff343a37ae35fd19 WatchSource:0}: Error finding container 07faf6fba47a0d054acd19a24a7b84a03664c9a82aeb4af4ff343a37ae35fd19: Status 404 returned error can't find the container with id 07faf6fba47a0d054acd19a24a7b84a03664c9a82aeb4af4ff343a37ae35fd19 Sep 16 05:00:30.775343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:30.775249 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" event={"ID":"abcf1429-ec8b-4377-aa14-438236a2df38","Type":"ContainerStarted","Data":"07faf6fba47a0d054acd19a24a7b84a03664c9a82aeb4af4ff343a37ae35fd19"} Sep 16 05:00:31.782059 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:31.782020 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" event={"ID":"abcf1429-ec8b-4377-aa14-438236a2df38","Type":"ContainerStarted","Data":"72d75d4d9e176637579d4971db524fa673352d272e5f0e57ef50784e8cb01f1a"} Sep 16 05:00:31.802502 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:31.802446 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" podStartSLOduration=1.674511879 podStartE2EDuration="2.802428783s" totalImagesPullingTime="1.127916904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:00:31.799959563 +0000 UTC m=+697.313511098" watchObservedRunningTime="2026-09-16 05:00:31.802428783 +0000 UTC m=+697.315980317" Sep 16 05:00:40.169457 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:40.169417 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:40.169979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:40.169487 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:40.182338 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:40.182312 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:40.849039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:40.849008 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc" Sep 16 05:00:55.153365 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.153330 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx"] Sep 16 05:00:55.156939 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.156919 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.159830 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.159812 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 16 05:00:55.167287 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.167265 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx"] Sep 16 05:00:55.297958 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.297930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.298106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.297964 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.298106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.298010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.298106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.298033 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqjn\" (UniqueName: \"kubernetes.io/projected/ade66949-a4fe-4ffe-801a-c5c56ba3303e-kube-api-access-gjqjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.298106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.298056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.298106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.298088 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.398840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.398803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.398871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.398913 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.398940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.399005 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399256 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.399133 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqjn\" (UniqueName: \"kubernetes.io/projected/ade66949-a4fe-4ffe-801a-c5c56ba3303e-kube-api-access-gjqjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399319 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.399257 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399319 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.399302 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.399419 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.399315 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.401293 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.401263 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade66949-a4fe-4ffe-801a-c5c56ba3303e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.401512 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.401495 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade66949-a4fe-4ffe-801a-c5c56ba3303e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.406912 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.406856 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqjn\" (UniqueName: \"kubernetes.io/projected/ade66949-a4fe-4ffe-801a-c5c56ba3303e-kube-api-access-gjqjn\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qq9jx\" (UID: \"ade66949-a4fe-4ffe-801a-c5c56ba3303e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.467896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.467863 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:00:55.753181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.753154 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx"] Sep 16 05:00:55.755087 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:00:55.755056 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade66949_a4fe_4ffe_801a_c5c56ba3303e.slice/crio-ff571ef3b764dc404193fb2edd505c3be681953ad23d7113c793b2959ff0d7d2 WatchSource:0}: Error finding container ff571ef3b764dc404193fb2edd505c3be681953ad23d7113c793b2959ff0d7d2: Status 404 returned error can't find the container with id ff571ef3b764dc404193fb2edd505c3be681953ad23d7113c793b2959ff0d7d2 Sep 16 05:00:55.919541 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:55.919510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" event={"ID":"ade66949-a4fe-4ffe-801a-c5c56ba3303e","Type":"ContainerStarted","Data":"ff571ef3b764dc404193fb2edd505c3be681953ad23d7113c793b2959ff0d7d2"} Sep 16 05:00:56.931017 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:56.930981 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" event={"ID":"ade66949-a4fe-4ffe-801a-c5c56ba3303e","Type":"ContainerStarted","Data":"ce2a2c53da24011f839d49ddf337580a2e3e22ae58cadf759748b9ceaa6e76d3"} Sep 16 05:00:56.949168 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:00:56.949122 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" podStartSLOduration=1.739288502 podStartE2EDuration="1.94910504s" totalImagesPullingTime="209.816538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:00:56.947578989 +0000 UTC m=+722.461130524" watchObservedRunningTime="2026-09-16 05:00:56.94910504 +0000 UTC m=+722.462656575" Sep 16 05:01:05.468693 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:01:05.468650 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:01:05.469316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:01:05.468719 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:01:05.481228 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:01:05.481203 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:01:05.992561 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:01:05.992515 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qq9jx" Sep 16 05:02:29.776767 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.776680 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86d444b47b-957qw"] Sep 16 05:02:29.780137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.780122 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.788364 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.788337 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d444b47b-957qw"] Sep 16 05:02:29.826659 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.826622 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-tls-cert\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.826823 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.826754 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxqs\" (UniqueName: \"kubernetes.io/projected/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-kube-api-access-hcxqs\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.928164 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.928126 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-tls-cert\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.928362 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.928253 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxqs\" (UniqueName: \"kubernetes.io/projected/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-kube-api-access-hcxqs\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.930704 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.930680 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-tls-cert\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:29.937137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:29.937117 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxqs\" (UniqueName: \"kubernetes.io/projected/b3e80c37-f0da-44b9-af00-f4fe2265e8d4-kube-api-access-hcxqs\") pod \"authorino-86d444b47b-957qw\" (UID: \"b3e80c37-f0da-44b9-af00-f4fe2265e8d4\") " pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:30.090388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:30.090360 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86d444b47b-957qw" Sep 16 05:02:30.361395 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:30.361370 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86d444b47b-957qw"] Sep 16 05:02:30.363592 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:30.363543 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3e80c37_f0da_44b9_af00_f4fe2265e8d4.slice/crio-c3120b84a405b79bcaf330c49db58f3ebd962fc94d881013cb989526facfb8f3 WatchSource:0}: Error finding container c3120b84a405b79bcaf330c49db58f3ebd962fc94d881013cb989526facfb8f3: Status 404 returned error can't find the container with id c3120b84a405b79bcaf330c49db58f3ebd962fc94d881013cb989526facfb8f3 Sep 16 05:02:30.449828 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:30.449793 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d444b47b-957qw" event={"ID":"b3e80c37-f0da-44b9-af00-f4fe2265e8d4","Type":"ContainerStarted","Data":"c3120b84a405b79bcaf330c49db58f3ebd962fc94d881013cb989526facfb8f3"} Sep 16 05:02:31.460425 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.460382 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86d444b47b-957qw" event={"ID":"b3e80c37-f0da-44b9-af00-f4fe2265e8d4","Type":"ContainerStarted","Data":"d4f4b70188f6f0a1b25a5f0f4e961eeebb0a3735043a9700d131493251405e2c"} Sep 16 05:02:31.480346 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.480276 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86d444b47b-957qw" podStartSLOduration=1.861234938 podStartE2EDuration="2.480253016s" totalImagesPullingTime="619.018078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:31.478218183 +0000 UTC m=+816.991769727" watchObservedRunningTime="2026-09-16 05:02:31.480253016 +0000 UTC m=+816.993804551" Sep 16 05:02:31.523265 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.522425 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 05:02:31.523265 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.522676 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" podUID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" containerName="authorino" containerID="cri-o://7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a" gracePeriod=30 Sep 16 05:02:31.780957 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.780926 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 05:02:31.847239 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.847196 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert\") pod \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " Sep 16 05:02:31.847239 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.847243 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8nr\" (UniqueName: \"kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr\") pod \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\" (UID: \"2cf67c98-186e-4b5c-80d7-eac7fda3974e\") " Sep 16 05:02:31.849345 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.849310 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr" (OuterVolumeSpecName: "kube-api-access-xp8nr") pod "2cf67c98-186e-4b5c-80d7-eac7fda3974e" (UID: "2cf67c98-186e-4b5c-80d7-eac7fda3974e"). InnerVolumeSpecName "kube-api-access-xp8nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:02:31.859106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.859076 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2cf67c98-186e-4b5c-80d7-eac7fda3974e" (UID: "2cf67c98-186e-4b5c-80d7-eac7fda3974e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:02:31.948049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.948005 2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2cf67c98-186e-4b5c-80d7-eac7fda3974e-tls-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:02:31.948049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:31.948047 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8nr\" (UniqueName: \"kubernetes.io/projected/2cf67c98-186e-4b5c-80d7-eac7fda3974e-kube-api-access-xp8nr\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:02:32.468740 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.468699 2560 generic.go:356] "Generic (PLEG): container finished" podID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" containerID="7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a" exitCode=0 Sep 16 05:02:32.469129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.468747 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" Sep 16 05:02:32.469129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.468777 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" event={"ID":"2cf67c98-186e-4b5c-80d7-eac7fda3974e","Type":"ContainerDied","Data":"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a"} Sep 16 05:02:32.469129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.468812 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77985d8bf7-wr8jg" event={"ID":"2cf67c98-186e-4b5c-80d7-eac7fda3974e","Type":"ContainerDied","Data":"1c8cb8e9bdf9d0489f9b12a8cfb45b35f04a01a0b4ef41b5f136011036c00601"} Sep 16 05:02:32.469129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.468835 2560 scope.go:117] "RemoveContainer" containerID="7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a" Sep 16 05:02:32.478759 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.478741 2560 scope.go:117] "RemoveContainer" containerID="7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a" Sep 16 05:02:32.479039 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:02:32.479016 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a\": container with ID starting with 7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a not found: ID does not exist" containerID="7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a" Sep 16 05:02:32.479091 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.479048 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a"} err="failed to get container status \"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a\": rpc error: code = NotFound desc = could not find container \"7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a\": container with ID starting with 7b2f2dcb73c5ff1231255bad4809f88a95454f5bcf883f102356c3dcc9d2381a not found: ID does not exist" Sep 16 05:02:32.496539 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.496501 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 05:02:32.500504 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:32.500475 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77985d8bf7-wr8jg"] Sep 16 05:02:33.121486 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:33.121450 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" path="/var/lib/kubelet/pods/2cf67c98-186e-4b5c-80d7-eac7fda3974e/volumes" Sep 16 05:02:48.763309 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.763278 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:02:48.763810 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.763654 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" containerName="authorino" Sep 16 05:02:48.763810 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.763665 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" containerName="authorino" Sep 16 05:02:48.763810 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.763748 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf67c98-186e-4b5c-80d7-eac7fda3974e" containerName="authorino" Sep 16 05:02:48.767773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.767750 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.770917 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.770896 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ee7a60-openshift-default-dockercfg-426f5\"" Sep 16 05:02:48.771047 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.770937 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 05:02:48.778383 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.778358 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:02:48.901980 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.901951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.901980 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.901986 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902192 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902192 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902192 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902192 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902208 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86q5x\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902283 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:48.902397 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:48.902310 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003386 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003351 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003386 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003416 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003435 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003454 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003477 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.003648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003588 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86q5x\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.004006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003859 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.004006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.003878 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.004112 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.004065 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.004300 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.004272 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.004300 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.004289 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.005892 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.005867 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.005983 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.005969 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.011343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.011312 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.011614 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.011598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86q5x\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x\") pod \"e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.079353 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.079315 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:49.359676 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.359642 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:02:49.361393 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:49.361367 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6dd537_7a92_4d81_8277_de36eafb5dfb.slice/crio-5c538ea2379ee6272085b7d1932d405c75775fbf173171b81e720e2e7d3ff84a WatchSource:0}: Error finding container 5c538ea2379ee6272085b7d1932d405c75775fbf173171b81e720e2e7d3ff84a: Status 404 returned error can't find the container with id 5c538ea2379ee6272085b7d1932d405c75775fbf173171b81e720e2e7d3ff84a Sep 16 05:02:49.568434 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:49.568399 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerStarted","Data":"5c538ea2379ee6272085b7d1932d405c75775fbf173171b81e720e2e7d3ff84a"} Sep 16 05:02:50.322231 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.322199 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7856-667dd2"] Sep 16 05:02:50.327029 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.327006 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:50.329727 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.329708 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fr2c9\"" Sep 16 05:02:50.334463 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.334440 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7856-667dd2"] Sep 16 05:02:50.417896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.417861 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54fbs\" (UniqueName: \"kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs\") pod \"test-curl-7856-667dd2\" (UID: \"47700284-6f9c-4ef3-9c90-dc5ec7713981\") " pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:50.518992 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.518958 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54fbs\" (UniqueName: \"kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs\") pod \"test-curl-7856-667dd2\" (UID: \"47700284-6f9c-4ef3-9c90-dc5ec7713981\") " pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:50.526808 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.526786 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54fbs\" (UniqueName: \"kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs\") pod \"test-curl-7856-667dd2\" (UID: \"47700284-6f9c-4ef3-9c90-dc5ec7713981\") " pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:50.638836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.638800 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:50.921423 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:50.921390 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7856-667dd2"] Sep 16 05:02:50.924054 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:50.924016 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47700284_6f9c_4ef3_9c90_dc5ec7713981.slice/crio-a43ec7159b20fd3fe90d68cf82d53ffb39013ab993c206a57767da37e9b1ff80 WatchSource:0}: Error finding container a43ec7159b20fd3fe90d68cf82d53ffb39013ab993c206a57767da37e9b1ff80: Status 404 returned error can't find the container with id a43ec7159b20fd3fe90d68cf82d53ffb39013ab993c206a57767da37e9b1ff80 Sep 16 05:02:51.584085 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:51.584047 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7856-667dd2" event={"ID":"47700284-6f9c-4ef3-9c90-dc5ec7713981","Type":"ContainerStarted","Data":"a43ec7159b20fd3fe90d68cf82d53ffb39013ab993c206a57767da37e9b1ff80"} Sep 16 05:02:53.478049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.478011 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:53.478360 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.478089 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:53.478360 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.478138 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:53.600692 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.600654 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerStarted","Data":"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5"} Sep 16 05:02:53.603163 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.603130 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7856-667dd2" event={"ID":"47700284-6f9c-4ef3-9c90-dc5ec7713981","Type":"ContainerStarted","Data":"fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4"} Sep 16 05:02:53.628157 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.628102 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podStartSLOduration=1.514156893 podStartE2EDuration="5.62808559s" totalImagesPullingTime="4.113928697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:53.625629889 +0000 UTC m=+839.139181449" watchObservedRunningTime="2026-09-16 05:02:53.62808559 +0000 UTC m=+839.141637124" Sep 16 05:02:53.646987 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:53.646933 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7856-667dd2" podStartSLOduration=1.095541308 podStartE2EDuration="3.646915766s" totalImagesPullingTime="2.551374458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:53.644396047 +0000 UTC m=+839.157947585" watchObservedRunningTime="2026-09-16 05:02:53.646915766 +0000 UTC m=+839.160467299" Sep 16 05:02:54.080094 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.080056 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:54.085418 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.085396 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:54.256350 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.256313 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7856-667dd2"] Sep 16 05:02:54.260263 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.260232 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7856-667dd2"] Sep 16 05:02:54.386291 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.386206 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7856-d04149"] Sep 16 05:02:54.386638 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.386623 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" containerName="test-curl-7856-667dd2" Sep 16 05:02:54.386691 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.386639 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" containerName="test-curl-7856-667dd2" Sep 16 05:02:54.386750 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.386741 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" containerName="test-curl-7856-667dd2" Sep 16 05:02:54.390055 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.390035 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-d04149" Sep 16 05:02:54.401792 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.401768 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7856-d04149"] Sep 16 05:02:54.459375 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.459330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8q6v\" (UniqueName: \"kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v\") pod \"test-curl-7856-d04149\" (UID: \"d2eb0137-c38b-4334-8387-42b52cfae4d6\") " pod="opendatahub/test-curl-7856-d04149" Sep 16 05:02:54.560506 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.560473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8q6v\" (UniqueName: \"kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v\") pod \"test-curl-7856-d04149\" (UID: \"d2eb0137-c38b-4334-8387-42b52cfae4d6\") " pod="opendatahub/test-curl-7856-d04149" Sep 16 05:02:54.568616 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.568582 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8q6v\" (UniqueName: \"kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v\") pod \"test-curl-7856-d04149\" (UID: \"d2eb0137-c38b-4334-8387-42b52cfae4d6\") " pod="opendatahub/test-curl-7856-d04149" Sep 16 05:02:54.609183 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:02:54.609147 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7856-667dd2" container="test-curl-7856-667dd2" Sep 16 05:02:54.609358 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.609234 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:54.610096 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.610079 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:02:54.699027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.698945 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-d04149" Sep 16 05:02:54.814119 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.814086 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:02:54.818804 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.818784 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:54.821582 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.821540 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-12b6c6\"" Sep 16 05:02:54.821773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.821755 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-12b6c6-dockercfg-9qhmh\"" Sep 16 05:02:54.827867 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.827834 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:02:54.965257 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.965177 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:54.965257 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.965214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcfl\" (UniqueName: \"kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:54.973792 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:54.973766 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7856-d04149"] Sep 16 05:02:54.975462 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:54.975435 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2eb0137_c38b_4334_8387_42b52cfae4d6.slice/crio-a95242adfdfdf65b4fbe7e6ac00cd52d9fb87fdfc5f2d1c77e2ae2f7c7369225 WatchSource:0}: Error finding container a95242adfdfdf65b4fbe7e6ac00cd52d9fb87fdfc5f2d1c77e2ae2f7c7369225: Status 404 returned error can't find the container with id a95242adfdfdf65b4fbe7e6ac00cd52d9fb87fdfc5f2d1c77e2ae2f7c7369225 Sep 16 05:02:55.066060 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.066034 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:55.066297 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.066072 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcfl\" (UniqueName: \"kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:55.070458 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.070440 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-12b6c6\"" Sep 16 05:02:55.077089 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.077070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:55.079640 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.079621 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcfl\" (UniqueName: \"kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl\") pod \"payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:55.133219 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.133196 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-12b6c6-dockercfg-9qhmh\"" Sep 16 05:02:55.141711 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.141688 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:55.415248 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.415215 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:02:55.418315 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:55.418279 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432e564f_441d_4647_a189_56df9c56fb04.slice/crio-0fa90dae4109f528b21bc40799d7b907c959a3a6aecd0b23bcef8d0908a3ab37 WatchSource:0}: Error finding container 0fa90dae4109f528b21bc40799d7b907c959a3a6aecd0b23bcef8d0908a3ab37: Status 404 returned error can't find the container with id 0fa90dae4109f528b21bc40799d7b907c959a3a6aecd0b23bcef8d0908a3ab37 Sep 16 05:02:55.616884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.616843 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7856-d04149" event={"ID":"d2eb0137-c38b-4334-8387-42b52cfae4d6","Type":"ContainerStarted","Data":"36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17"} Sep 16 05:02:55.616884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.616888 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7856-d04149" event={"ID":"d2eb0137-c38b-4334-8387-42b52cfae4d6","Type":"ContainerStarted","Data":"a95242adfdfdf65b4fbe7e6ac00cd52d9fb87fdfc5f2d1c77e2ae2f7c7369225"} Sep 16 05:02:55.618124 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.618094 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" event={"ID":"432e564f-441d-4647-a189-56df9c56fb04","Type":"ContainerStarted","Data":"0fa90dae4109f528b21bc40799d7b907c959a3a6aecd0b23bcef8d0908a3ab37"} Sep 16 05:02:55.618350 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.618310 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7856-667dd2" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" containerName="test-curl-7856-667dd2" containerID="cri-o://fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4" gracePeriod=2 Sep 16 05:02:55.632722 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:55.632674 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7856-d04149" podStartSLOduration=1.632656301 podStartE2EDuration="1.632656301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:55.631194005 +0000 UTC m=+841.144745541" watchObservedRunningTime="2026-09-16 05:02:55.632656301 +0000 UTC m=+841.146207835" Sep 16 05:02:56.261593 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:56.261557 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7856-d04149"] Sep 16 05:02:56.265938 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:56.265908 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7856-d04149"] Sep 16 05:02:56.626605 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:56.626560 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" event={"ID":"432e564f-441d-4647-a189-56df9c56fb04","Type":"ContainerStarted","Data":"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76"} Sep 16 05:02:56.627015 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:56.626629 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:02:56.627015 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:02:56.626803 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7856-d04149" container="test-curl-7856-d04149" Sep 16 05:02:56.662388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:56.662340 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" podStartSLOduration=1.5797021629999999 podStartE2EDuration="2.66232324s" totalImagesPullingTime="1.082621077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:56.659646652 +0000 UTC m=+842.173198186" watchObservedRunningTime="2026-09-16 05:02:56.66232324 +0000 UTC m=+842.175874792" Sep 16 05:02:57.075956 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.075914 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:02:57.076459 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.076441 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" containerName="test-curl-7856-d04149" Sep 16 05:02:57.076532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.076462 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" containerName="test-curl-7856-d04149" Sep 16 05:02:57.076532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.076529 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" containerName="test-curl-7856-d04149" Sep 16 05:02:57.079719 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.079702 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.082134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.082114 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d05eb5-openshift-default-dockercfg-98snj\"" Sep 16 05:02:57.091598 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.091573 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:02:57.184634 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184634 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfsmc\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184713 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.184896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184867 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.185222 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.184904 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286452 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286412 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286645 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286470 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286645 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286645 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286606 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286658 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286724 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfsmc\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286789 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.286831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.287106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286877 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.287106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.286923 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.287217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.287092 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.287274 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.287233 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.287613 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.287593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.289075 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.289055 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.289458 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.289436 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.294669 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.294644 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.295091 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.295069 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfsmc\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc\") pod \"e2e-autores-d05eb5-openshift-default-7875475df6-n682j\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.391349 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.391258 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:57.634493 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.634452 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7856-d04149" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" containerName="test-curl-7856-d04149" containerID="cri-o://36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17" gracePeriod=2 Sep 16 05:02:57.678886 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.678847 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:02:57.707708 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:57.707670 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60aa236f_12ed_40fc_92fe_d67bc473f3fc.slice/crio-e99eb6212f4cace9a0a23cbe7a2b1c22835188cc6bcce5beadbcc278b7fc6c4f WatchSource:0}: Error finding container e99eb6212f4cace9a0a23cbe7a2b1c22835188cc6bcce5beadbcc278b7fc6c4f: Status 404 returned error can't find the container with id e99eb6212f4cace9a0a23cbe7a2b1c22835188cc6bcce5beadbcc278b7fc6c4f Sep 16 05:02:57.710035 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.710002 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:57.710122 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.710070 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:57.710122 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.710097 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:02:57.793644 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.793618 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:57.892688 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.892653 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54fbs\" (UniqueName: \"kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs\") pod \"47700284-6f9c-4ef3-9c90-dc5ec7713981\" (UID: \"47700284-6f9c-4ef3-9c90-dc5ec7713981\") " Sep 16 05:02:57.894966 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.894936 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs" (OuterVolumeSpecName: "kube-api-access-54fbs") pod "47700284-6f9c-4ef3-9c90-dc5ec7713981" (UID: "47700284-6f9c-4ef3-9c90-dc5ec7713981"). InnerVolumeSpecName "kube-api-access-54fbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:02:57.993957 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:57.993884 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54fbs\" (UniqueName: \"kubernetes.io/projected/47700284-6f9c-4ef3-9c90-dc5ec7713981-kube-api-access-54fbs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:02:58.236934 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.236900 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:02:58.240433 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.240408 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.243222 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.243199 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ee7a60\"" Sep 16 05:02:58.243363 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.243327 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ee7a60-dockercfg-q56s8\"" Sep 16 05:02:58.252188 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.252167 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:02:58.397692 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.397650 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdt9\" (UniqueName: \"kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.397856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.397731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.499127 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.499040 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdt9\" (UniqueName: \"kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.499127 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.499122 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.499919 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.499897 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.508234 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.508206 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdt9\" (UniqueName: \"kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9\") pod \"payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.551881 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.551841 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:58.644094 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.644053 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerStarted","Data":"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c"} Sep 16 05:02:58.644527 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.644113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerStarted","Data":"e99eb6212f4cace9a0a23cbe7a2b1c22835188cc6bcce5beadbcc278b7fc6c4f"} Sep 16 05:02:58.645434 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.645412 2560 generic.go:356] "Generic (PLEG): container finished" podID="47700284-6f9c-4ef3-9c90-dc5ec7713981" containerID="fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4" exitCode=137 Sep 16 05:02:58.645581 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.645450 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:58.645581 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.645483 2560 scope.go:117] "RemoveContainer" containerID="fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4" Sep 16 05:02:58.655442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.655424 2560 scope.go:117] "RemoveContainer" containerID="fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4" Sep 16 05:02:58.655767 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:02:58.655749 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4\": container with ID starting with fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4 not found: ID does not exist" containerID="fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4" Sep 16 05:02:58.655827 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.655776 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4"} err="failed to get container status \"fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4\": rpc error: code = NotFound desc = could not find container \"fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4\": container with ID starting with fe366ace2c2083ae1e884493b146fa16a8e924ee6ab96db2bcc84dc1aa85c2c4 not found: ID does not exist" Sep 16 05:02:58.663261 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.663222 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podStartSLOduration=1.663208505 podStartE2EDuration="1.663208505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:58.662002723 +0000 UTC m=+844.175554269" watchObservedRunningTime="2026-09-16 05:02:58.663208505 +0000 UTC m=+844.176760039" Sep 16 05:02:58.664264 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:02:58.664240 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7856-667dd2\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" pod="opendatahub/test-curl-7856-667dd2" Sep 16 05:02:58.835260 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:58.835230 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:02:58.836267 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:02:58.836241 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa50c0ce_6901_4ae5_bfc3_7e94d0b86cc6.slice/crio-b43ea2f4e542b4843a24ea3f303e4496b3cf14951837e50b46ef1d8b82ce07cd WatchSource:0}: Error finding container b43ea2f4e542b4843a24ea3f303e4496b3cf14951837e50b46ef1d8b82ce07cd: Status 404 returned error can't find the container with id b43ea2f4e542b4843a24ea3f303e4496b3cf14951837e50b46ef1d8b82ce07cd Sep 16 05:02:59.121475 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.121382 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47700284-6f9c-4ef3-9c90-dc5ec7713981" path="/var/lib/kubelet/pods/47700284-6f9c-4ef3-9c90-dc5ec7713981/volumes" Sep 16 05:02:59.392275 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.392197 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:59.397847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.397816 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:59.654064 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.654023 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" event={"ID":"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6","Type":"ContainerStarted","Data":"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72"} Sep 16 05:02:59.654459 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.654094 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" event={"ID":"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6","Type":"ContainerStarted","Data":"b43ea2f4e542b4843a24ea3f303e4496b3cf14951837e50b46ef1d8b82ce07cd"} Sep 16 05:02:59.654459 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.654210 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:02:59.654608 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.654580 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:59.655582 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.655383 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:02:59.670806 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.670745 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" podStartSLOduration=1.67072625 podStartE2EDuration="1.67072625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:02:59.669619369 +0000 UTC m=+845.183170904" watchObservedRunningTime="2026-09-16 05:02:59.67072625 +0000 UTC m=+845.184277785" Sep 16 05:02:59.859129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:02:59.859103 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-d04149" Sep 16 05:03:00.016682 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.016584 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8q6v\" (UniqueName: \"kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v\") pod \"d2eb0137-c38b-4334-8387-42b52cfae4d6\" (UID: \"d2eb0137-c38b-4334-8387-42b52cfae4d6\") " Sep 16 05:03:00.018841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.018803 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v" (OuterVolumeSpecName: "kube-api-access-n8q6v") pod "d2eb0137-c38b-4334-8387-42b52cfae4d6" (UID: "d2eb0137-c38b-4334-8387-42b52cfae4d6"). InnerVolumeSpecName "kube-api-access-n8q6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:03:00.118351 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.118317 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8q6v\" (UniqueName: \"kubernetes.io/projected/d2eb0137-c38b-4334-8387-42b52cfae4d6-kube-api-access-n8q6v\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:03:00.662207 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.662172 2560 generic.go:356] "Generic (PLEG): container finished" podID="d2eb0137-c38b-4334-8387-42b52cfae4d6" containerID="36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17" exitCode=137 Sep 16 05:03:00.662693 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.662219 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7856-d04149" Sep 16 05:03:00.662693 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.662272 2560 scope.go:117] "RemoveContainer" containerID="36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17" Sep 16 05:03:00.673134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.673111 2560 scope.go:117] "RemoveContainer" containerID="36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17" Sep 16 05:03:00.673458 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:03:00.673439 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17\": container with ID starting with 36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17 not found: ID does not exist" containerID="36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17" Sep 16 05:03:00.673523 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:00.673467 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17"} err="failed to get container status \"36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17\": rpc error: code = NotFound desc = could not find container \"36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17\": container with ID starting with 36c74a47ccb4e8b3808f5f38a826fff258c63ba069e4823c106d5edf0aee3f17 not found: ID does not exist" Sep 16 05:03:00.674034 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:03:00.674014 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7856-d04149\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" pod="opendatahub/test-curl-7856-d04149" Sep 16 05:03:01.121529 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:01.121491 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2eb0137-c38b-4334-8387-42b52cfae4d6" path="/var/lib/kubelet/pods/d2eb0137-c38b-4334-8387-42b52cfae4d6/volumes" Sep 16 05:03:03.332311 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.332281 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:03:03.337309 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.337287 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.339765 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.339742 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-d05eb5-serving-cert\"" Sep 16 05:03:03.351236 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.351210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:03:03.455391 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.455356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6q9q\" (UniqueName: \"kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.455391 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.455403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.455663 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.455536 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.455728 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.455668 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.557010 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.556974 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.557178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.557039 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6q9q\" (UniqueName: \"kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.557178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.557071 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.557178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.557173 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.557365 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:03:03.557349 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-serving-cert: secret "maas-api-e2e-autores-d05eb5-serving-cert" not found Sep 16 05:03:03.557433 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:03:03.557419 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls podName:603da56e-c461-4027-90ea-4434357abee9 nodeName:}" failed. No retries permitted until 2026-09-16 05:03:04.05739822 +0000 UTC m=+849.570949746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls") pod "maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" (UID: "603da56e-c461-4027-90ea-4434357abee9") : secret "maas-api-e2e-autores-d05eb5-serving-cert" not found Sep 16 05:03:03.557836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.557816 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.559593 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.559573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.565769 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.565742 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6q9q\" (UniqueName: \"kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:03.925829 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.925787 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:03:03.929722 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.929699 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:03.932627 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.932602 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d05eb5\"" Sep 16 05:03:03.932749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.932704 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d05eb5-dockercfg-s2kcf\"" Sep 16 05:03:03.939132 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:03.939069 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:03:04.061663 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.061635 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:04.061856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.061686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8f2\" (UniqueName: \"kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.061856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.061817 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.064405 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.064373 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") pod \"maas-api-e2e-autores-d05eb5-587fcf96-pj7t9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:04.163129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.163089 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.163326 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.163156 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8f2\" (UniqueName: \"kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.163862 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.163837 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.171255 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.171226 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8f2\" (UniqueName: \"kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2\") pod \"payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.241587 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.241498 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.252593 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.252538 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:04.316858 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.316761 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:04.323013 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.322989 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.326054 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.325786 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b9f9e7ea-serving-cert\"" Sep 16 05:03:04.332513 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.332490 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:04.466082 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.466049 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.466082 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.466082 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs824\" (UniqueName: \"kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.466287 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.466178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.466287 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.466231 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.522923 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.522899 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:03:04.524200 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:03:04.524172 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0a5d26_e818_428b_9837_4ac87536de07.slice/crio-b36f5ee47c709ce59185ab6ad4c587f4e660f9a21f4742ffd1272a34e6e36c3a WatchSource:0}: Error finding container b36f5ee47c709ce59185ab6ad4c587f4e660f9a21f4742ffd1272a34e6e36c3a: Status 404 returned error can't find the container with id b36f5ee47c709ce59185ab6ad4c587f4e660f9a21f4742ffd1272a34e6e36c3a Sep 16 05:03:04.556232 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.556208 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:03:04.566649 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:03:04.566623 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603da56e_c461_4027_90ea_4434357abee9.slice/crio-3235cc20f49a57d3f101312e4a50da809d0b6f70d794f3e819b150874b7a86e0 WatchSource:0}: Error finding container 3235cc20f49a57d3f101312e4a50da809d0b6f70d794f3e819b150874b7a86e0: Status 404 returned error can't find the container with id 3235cc20f49a57d3f101312e4a50da809d0b6f70d794f3e819b150874b7a86e0 Sep 16 05:03:04.566834 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.566811 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.566909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.566871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.566909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.566900 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs824\" (UniqueName: \"kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.567054 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.567036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.567505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.567481 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.569868 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.569840 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.570065 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.570027 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.575261 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.575234 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs824\" (UniqueName: \"kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824\") pod \"maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.641339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.641309 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:04.693820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.693741 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" event={"ID":"1c0a5d26-e818-428b-9837-4ac87536de07","Type":"ContainerStarted","Data":"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075"} Sep 16 05:03:04.693820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.693810 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" event={"ID":"1c0a5d26-e818-428b-9837-4ac87536de07","Type":"ContainerStarted","Data":"b36f5ee47c709ce59185ab6ad4c587f4e660f9a21f4742ffd1272a34e6e36c3a"} Sep 16 05:03:04.694211 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.694190 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:04.696194 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.696141 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" event={"ID":"603da56e-c461-4027-90ea-4434357abee9","Type":"ContainerStarted","Data":"3235cc20f49a57d3f101312e4a50da809d0b6f70d794f3e819b150874b7a86e0"} Sep 16 05:03:04.712849 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.712789 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" podStartSLOduration=1.712765719 podStartE2EDuration="1.712765719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:04.712307491 +0000 UTC m=+850.225859024" watchObservedRunningTime="2026-09-16 05:03:04.712765719 +0000 UTC m=+850.226317253" Sep 16 05:03:04.925389 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:04.925356 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:04.926964 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:03:04.926913 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc010fba1_2a45_431b_a106_dc37a13b5df2.slice/crio-051a636adfc1f2b9250fdc54b7d574f09518d5f464644df88ed8b5a838f9d85c WatchSource:0}: Error finding container 051a636adfc1f2b9250fdc54b7d574f09518d5f464644df88ed8b5a838f9d85c: Status 404 returned error can't find the container with id 051a636adfc1f2b9250fdc54b7d574f09518d5f464644df88ed8b5a838f9d85c Sep 16 05:03:05.703844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.703791 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" event={"ID":"603da56e-c461-4027-90ea-4434357abee9","Type":"ContainerStarted","Data":"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f"} Sep 16 05:03:05.704249 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.703866 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:05.705479 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.705456 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" event={"ID":"c010fba1-2a45-431b-a106-dc37a13b5df2","Type":"ContainerStarted","Data":"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63"} Sep 16 05:03:05.705604 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.705496 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" event={"ID":"c010fba1-2a45-431b-a106-dc37a13b5df2","Type":"ContainerStarted","Data":"051a636adfc1f2b9250fdc54b7d574f09518d5f464644df88ed8b5a838f9d85c"} Sep 16 05:03:05.705690 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.705668 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:05.723316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.723272 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" podStartSLOduration=2.31313702 podStartE2EDuration="2.723257103s" totalImagesPullingTime="410.120083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:05.720336065 +0000 UTC m=+851.233887622" watchObservedRunningTime="2026-09-16 05:03:05.723257103 +0000 UTC m=+851.236808636" Sep 16 05:03:05.738109 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:05.738068 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" podStartSLOduration=1.356481258 podStartE2EDuration="1.738057854s" totalImagesPullingTime="381.576596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:05.737585436 +0000 UTC m=+851.251136975" watchObservedRunningTime="2026-09-16 05:03:05.738057854 +0000 UTC m=+851.251609388" Sep 16 05:03:07.637163 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:07.637131 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:03:10.665831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:10.665789 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:03:11.716623 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:11.716595 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:03:11.717072 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:11.717013 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:15.708435 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:15.708404 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:03:19.816785 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.816735 2560 generic.go:356] "Generic (PLEG): container finished" podID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerID="960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5" exitCode=0 Sep 16 05:03:19.817309 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.816812 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerDied","Data":"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5"} Sep 16 05:03:19.817378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.817362 2560 scope.go:117] "RemoveContainer" containerID="960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5" Sep 16 05:03:19.818528 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.818498 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:19.818633 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.818581 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:19.818633 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:19.818616 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:20.228814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.228788 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:03:20.243342 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.242473 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.246048 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.246018 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:03:20.246956 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.246846 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d1f291-dockercfg-gfhw9\"" Sep 16 05:03:20.247186 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.247151 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d1f291\"" Sep 16 05:03:20.347313 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.347277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.347477 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.347445 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v72v\" (UniqueName: \"kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.448760 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.448727 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.448942 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.448798 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v72v\" (UniqueName: \"kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.449482 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.449460 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.457363 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.457337 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v72v\" (UniqueName: \"kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v\") pod \"payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.556299 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.556190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:20.827718 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.827677 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerStarted","Data":"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8"} Sep 16 05:03:20.830563 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.830515 2560 generic.go:356] "Generic (PLEG): container finished" podID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerID="f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c" exitCode=0 Sep 16 05:03:20.831526 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.830714 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerDied","Data":"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c"} Sep 16 05:03:20.831526 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.831164 2560 scope.go:117] "RemoveContainer" containerID="f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c" Sep 16 05:03:20.833175 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.833140 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:20.834339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.834310 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:20.834528 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.834503 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:03:20.839757 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:20.839644 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:03:21.081691 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.081593 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:03:21.081870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.081743 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:21.081870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.081787 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:21.838830 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.838791 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerStarted","Data":"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9"} Sep 16 05:03:21.840475 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.840448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" event={"ID":"b2637674-7a81-4cde-938e-361a28cfb006","Type":"ContainerStarted","Data":"b97ba219e723835280e4b018e7e811a4b9cdfc54edb94720d04e64c8f199b3d6"} Sep 16 05:03:21.840819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.840742 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:21.840819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.840779 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" event={"ID":"b2637674-7a81-4cde-938e-361a28cfb006","Type":"ContainerStarted","Data":"49cf25c613f1a749c0300f45bd17b16b930ee0f1ea2e888893a2caa45b5bb2c9"} Sep 16 05:03:21.878684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:21.878627 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" podStartSLOduration=1.878608093 podStartE2EDuration="1.878608093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:03:21.875379095 +0000 UTC m=+867.388930626" watchObservedRunningTime="2026-09-16 05:03:21.878608093 +0000 UTC m=+867.392159627" Sep 16 05:03:22.079946 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:22.079908 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:22.080245 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:22.080211 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:22.392706 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:22.391905 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:03:22.392706 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:22.392028 2560 patch_prober.go:28] interesting pod/e2e-autores-d05eb5-openshift-default-7875475df6-n682j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 16 05:03:22.392706 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:22.392073 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 16 05:03:23.080709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:23.080669 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:23.081346 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:23.081294 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:23.392467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:23.392360 2560 patch_prober.go:28] interesting pod/e2e-autores-d05eb5-openshift-default-7875475df6-n682j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 16 05:03:23.392467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:23.392433 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 16 05:03:24.080132 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:24.080066 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:24.080403 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:24.080370 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:24.392438 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:24.392343 2560 patch_prober.go:28] interesting pod/e2e-autores-d05eb5-openshift-default-7875475df6-n682j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 16 05:03:24.393019 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:24.392988 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 16 05:03:24.609107 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:24.609067 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:03:25.087579 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:25.082980 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:25.087579 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:25.083066 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:25.392154 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:25.392060 2560 patch_prober.go:28] interesting pod/e2e-autores-d05eb5-openshift-default-7875475df6-n682j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 16 05:03:25.392422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:25.392392 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 16 05:03:26.079769 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:26.079728 2560 patch_prober.go:28] interesting pod/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 16 05:03:26.080259 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:26.079808 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 16 05:03:26.392179 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:26.392078 2560 patch_prober.go:28] interesting pod/e2e-autores-d05eb5-openshift-default-7875475df6-n682j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 16 05:03:26.392179 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:26.392155 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 16 05:03:27.084424 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:27.084386 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:03:27.085580 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:27.085535 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:03:27.397187 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:27.397104 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:03:27.397544 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:27.397526 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:03:27.398408 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:27.398388 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:03:31.009153 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.009120 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:31.009651 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.009402 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" podUID="c010fba1-2a45-431b-a106-dc37a13b5df2" containerName="maas-api" containerID="cri-o://f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63" gracePeriod=30 Sep 16 05:03:31.271576 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.271505 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:31.374310 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374278 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca\") pod \"c010fba1-2a45-431b-a106-dc37a13b5df2\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " Sep 16 05:03:31.374510 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374352 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls\") pod \"c010fba1-2a45-431b-a106-dc37a13b5df2\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " Sep 16 05:03:31.374510 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374382 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs\") pod \"c010fba1-2a45-431b-a106-dc37a13b5df2\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " Sep 16 05:03:31.374510 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374474 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs824\" (UniqueName: \"kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824\") pod \"c010fba1-2a45-431b-a106-dc37a13b5df2\" (UID: \"c010fba1-2a45-431b-a106-dc37a13b5df2\") " Sep 16 05:03:31.374716 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374699 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c010fba1-2a45-431b-a106-dc37a13b5df2" (UID: "c010fba1-2a45-431b-a106-dc37a13b5df2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:03:31.374915 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.374888 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c010fba1-2a45-431b-a106-dc37a13b5df2-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:03:31.377218 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.377190 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c010fba1-2a45-431b-a106-dc37a13b5df2" (UID: "c010fba1-2a45-431b-a106-dc37a13b5df2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:03:31.377324 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.377201 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824" (OuterVolumeSpecName: "kube-api-access-xs824") pod "c010fba1-2a45-431b-a106-dc37a13b5df2" (UID: "c010fba1-2a45-431b-a106-dc37a13b5df2"). InnerVolumeSpecName "kube-api-access-xs824". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:03:31.377324 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.377248 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c010fba1-2a45-431b-a106-dc37a13b5df2" (UID: "c010fba1-2a45-431b-a106-dc37a13b5df2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:03:31.476049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.476014 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:03:31.476049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.476045 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c010fba1-2a45-431b-a106-dc37a13b5df2-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:03:31.476049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.476055 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs824\" (UniqueName: \"kubernetes.io/projected/c010fba1-2a45-431b-a106-dc37a13b5df2-kube-api-access-xs824\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:03:31.927727 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.927690 2560 generic.go:356] "Generic (PLEG): container finished" podID="c010fba1-2a45-431b-a106-dc37a13b5df2" containerID="f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63" exitCode=0 Sep 16 05:03:31.927904 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.927738 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" Sep 16 05:03:31.927904 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.927761 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" event={"ID":"c010fba1-2a45-431b-a106-dc37a13b5df2","Type":"ContainerDied","Data":"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63"} Sep 16 05:03:31.927904 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.927815 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf" event={"ID":"c010fba1-2a45-431b-a106-dc37a13b5df2","Type":"ContainerDied","Data":"051a636adfc1f2b9250fdc54b7d574f09518d5f464644df88ed8b5a838f9d85c"} Sep 16 05:03:31.927904 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.927844 2560 scope.go:117] "RemoveContainer" containerID="f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63" Sep 16 05:03:31.938427 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.938407 2560 scope.go:117] "RemoveContainer" containerID="f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63" Sep 16 05:03:31.938700 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:03:31.938678 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63\": container with ID starting with f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63 not found: ID does not exist" containerID="f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63" Sep 16 05:03:31.938754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.938709 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63"} err="failed to get container status \"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63\": rpc error: code = NotFound desc = could not find container \"f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63\": container with ID starting with f613a388cef8016d9e186795daa8eed67f818164fd37f6c8449e0ca93ca5db63 not found: ID does not exist" Sep 16 05:03:31.950166 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.950144 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:31.954794 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:31.954770 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9f9e7ea-5984977d78-5dsnf"] Sep 16 05:03:32.850938 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:32.850907 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:03:33.122477 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:33.122400 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c010fba1-2a45-431b-a106-dc37a13b5df2" path="/var/lib/kubelet/pods/c010fba1-2a45-431b-a106-dc37a13b5df2/volumes" Sep 16 05:03:54.990746 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:54.990719 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:03:54.993859 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:54.993841 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:03:54.996406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:54.996386 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:03:54.999298 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:03:54.999280 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:05:05.044519 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.044486 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:05:05.045076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.044984 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c010fba1-2a45-431b-a106-dc37a13b5df2" containerName="maas-api" Sep 16 05:05:05.045076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.044998 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c010fba1-2a45-431b-a106-dc37a13b5df2" containerName="maas-api" Sep 16 05:05:05.045155 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.045085 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c010fba1-2a45-431b-a106-dc37a13b5df2" containerName="maas-api" Sep 16 05:05:05.048194 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.048175 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.052018 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.051995 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d05eb5\"/\"default-dockercfg-mqj6r\"" Sep 16 05:05:05.052145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.052049 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d05eb5\"/\"kube-root-ca.crt\"" Sep 16 05:05:05.052145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.052079 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-d05eb5\"/\"autores-model-d05eb5-kserve-self-signed-certs\"" Sep 16 05:05:05.052145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.052109 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-d05eb5\"/\"openshift-service-ca.crt\"" Sep 16 05:05:05.058919 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.058898 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:05:05.129208 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkzs\" (UniqueName: \"kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.129384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129245 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.129384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129308 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.129384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129338 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.129384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129364 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.129532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.129427 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230507 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkzs\" (UniqueName: \"kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230605 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230645 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.230735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230669 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.231016 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.230967 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.231016 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.231005 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.231134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.231052 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.232868 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.232839 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.233086 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.233068 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.238359 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.238340 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkzs\" (UniqueName: \"kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs\") pod \"autores-model-d05eb5-kserve-76c88944d7-2hvrn\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.357754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.357728 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:05.644864 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:05.644839 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:05:05.647706 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:05:05.647679 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod782850a2_6484_4812_92e7_3cd82c649ca8.slice/crio-9d30acc5a2e7fb0a0e93460718d0b2b1a9e36174cf67298f59f9af8a84abe62f WatchSource:0}: Error finding container 9d30acc5a2e7fb0a0e93460718d0b2b1a9e36174cf67298f59f9af8a84abe62f: Status 404 returned error can't find the container with id 9d30acc5a2e7fb0a0e93460718d0b2b1a9e36174cf67298f59f9af8a84abe62f Sep 16 05:05:06.499176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:06.499144 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" event={"ID":"782850a2-6484-4812-92e7-3cd82c649ca8","Type":"ContainerStarted","Data":"62aa4ca36152470300641c9feb770bd121208c2e0de35207e0a4f6032bac6a09"} Sep 16 05:05:06.499176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:06.499183 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" event={"ID":"782850a2-6484-4812-92e7-3cd82c649ca8","Type":"ContainerStarted","Data":"9d30acc5a2e7fb0a0e93460718d0b2b1a9e36174cf67298f59f9af8a84abe62f"} Sep 16 05:05:06.518018 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:06.517952 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" podStartSLOduration=1.517936498 podStartE2EDuration="1.517936498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:05:06.517185736 +0000 UTC m=+972.030737266" watchObservedRunningTime="2026-09-16 05:05:06.517936498 +0000 UTC m=+972.031488031" Sep 16 05:05:15.357879 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:15.357846 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:15.358413 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:15.357903 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:15.374780 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:15.374753 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:15.571155 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:15.571130 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:05:47.653780 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:47.653751 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:05:49.771518 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:49.771487 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:05:49.771930 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:49.771752 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" podUID="603da56e-c461-4027-90ea-4434357abee9" containerName="maas-api" containerID="cri-o://4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f" gracePeriod=30 Sep 16 05:05:49.825400 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:49.825367 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:05:49.825628 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:49.825606 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" podUID="1c0a5d26-e818-428b-9837-4ac87536de07" containerName="payload-pre-processing" containerID="cri-o://4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075" gracePeriod=30 Sep 16 05:05:50.111263 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.111235 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:05:50.130048 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.129860 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:05:50.293446 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293364 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") pod \"603da56e-c461-4027-90ea-4434357abee9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " Sep 16 05:05:50.293446 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293406 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca\") pod \"603da56e-c461-4027-90ea-4434357abee9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " Sep 16 05:05:50.293446 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293435 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6q9q\" (UniqueName: \"kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q\") pod \"603da56e-c461-4027-90ea-4434357abee9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " Sep 16 05:05:50.293760 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293451 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume\") pod \"1c0a5d26-e818-428b-9837-4ac87536de07\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " Sep 16 05:05:50.293760 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293516 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs\") pod \"603da56e-c461-4027-90ea-4434357abee9\" (UID: \"603da56e-c461-4027-90ea-4434357abee9\") " Sep 16 05:05:50.293760 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293535 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cc8f2\" (UniqueName: \"kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2\") pod \"1c0a5d26-e818-428b-9837-4ac87536de07\" (UID: \"1c0a5d26-e818-428b-9837-4ac87536de07\") " Sep 16 05:05:50.293966 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.293938 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c0a5d26-e818-428b-9837-4ac87536de07" (UID: "1c0a5d26-e818-428b-9837-4ac87536de07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:05:50.294068 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.294034 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "603da56e-c461-4027-90ea-4434357abee9" (UID: "603da56e-c461-4027-90ea-4434357abee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:05:50.295728 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.295700 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "603da56e-c461-4027-90ea-4434357abee9" (UID: "603da56e-c461-4027-90ea-4434357abee9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:05:50.295834 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.295811 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2" (OuterVolumeSpecName: "kube-api-access-cc8f2") pod "1c0a5d26-e818-428b-9837-4ac87536de07" (UID: "1c0a5d26-e818-428b-9837-4ac87536de07"). InnerVolumeSpecName "kube-api-access-cc8f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:05:50.296050 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.296032 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "603da56e-c461-4027-90ea-4434357abee9" (UID: "603da56e-c461-4027-90ea-4434357abee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:05:50.296107 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.296047 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q" (OuterVolumeSpecName: "kube-api-access-t6q9q") pod "603da56e-c461-4027-90ea-4434357abee9" (UID: "603da56e-c461-4027-90ea-4434357abee9"). InnerVolumeSpecName "kube-api-access-t6q9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:05:50.395111 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395084 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6q9q\" (UniqueName: \"kubernetes.io/projected/603da56e-c461-4027-90ea-4434357abee9-kube-api-access-t6q9q\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.395111 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395110 2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0a5d26-e818-428b-9837-4ac87536de07-plugins-config-volume\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.395296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395121 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.395296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395129 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8f2\" (UniqueName: \"kubernetes.io/projected/1c0a5d26-e818-428b-9837-4ac87536de07-kube-api-access-cc8f2\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.395296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395138 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/603da56e-c461-4027-90ea-4434357abee9-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.395296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.395146 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/603da56e-c461-4027-90ea-4434357abee9-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:05:50.774794 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.774757 2560 generic.go:356] "Generic (PLEG): container finished" podID="1c0a5d26-e818-428b-9837-4ac87536de07" containerID="4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075" exitCode=0 Sep 16 05:05:50.775244 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.774813 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" Sep 16 05:05:50.775244 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.774833 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" event={"ID":"1c0a5d26-e818-428b-9837-4ac87536de07","Type":"ContainerDied","Data":"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075"} Sep 16 05:05:50.775244 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.774898 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv" event={"ID":"1c0a5d26-e818-428b-9837-4ac87536de07","Type":"ContainerDied","Data":"b36f5ee47c709ce59185ab6ad4c587f4e660f9a21f4742ffd1272a34e6e36c3a"} Sep 16 05:05:50.775244 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.774932 2560 scope.go:117] "RemoveContainer" containerID="4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075" Sep 16 05:05:50.776462 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.776442 2560 generic.go:356] "Generic (PLEG): container finished" podID="603da56e-c461-4027-90ea-4434357abee9" containerID="4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f" exitCode=0 Sep 16 05:05:50.776589 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.776501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" event={"ID":"603da56e-c461-4027-90ea-4434357abee9","Type":"ContainerDied","Data":"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f"} Sep 16 05:05:50.776589 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.776508 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" Sep 16 05:05:50.776589 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.776539 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9" event={"ID":"603da56e-c461-4027-90ea-4434357abee9","Type":"ContainerDied","Data":"3235cc20f49a57d3f101312e4a50da809d0b6f70d794f3e819b150874b7a86e0"} Sep 16 05:05:50.797270 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.797251 2560 scope.go:117] "RemoveContainer" containerID="4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075" Sep 16 05:05:50.797576 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:05:50.797540 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075\": container with ID starting with 4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075 not found: ID does not exist" containerID="4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075" Sep 16 05:05:50.797626 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.797585 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075"} err="failed to get container status \"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075\": rpc error: code = NotFound desc = could not find container \"4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075\": container with ID starting with 4fd8a7da761aaac7f6f660d13e8813ceed4dde84f92fc8bfe8f7758da6ce6075 not found: ID does not exist" Sep 16 05:05:50.797626 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.797602 2560 scope.go:117] "RemoveContainer" containerID="4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f" Sep 16 05:05:50.807620 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.807603 2560 scope.go:117] "RemoveContainer" containerID="4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f" Sep 16 05:05:50.807912 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:05:50.807894 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f\": container with ID starting with 4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f not found: ID does not exist" containerID="4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f" Sep 16 05:05:50.807971 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.807918 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f"} err="failed to get container status \"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f\": rpc error: code = NotFound desc = could not find container \"4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f\": container with ID starting with 4d9cf5f6fc4a42b469e368400ea12d08d56bc329e6828395f950b00d5eef198f not found: ID does not exist" Sep 16 05:05:50.810361 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.810342 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:05:50.817271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.817249 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d05eb5-587fcf96-pj7t9"] Sep 16 05:05:50.827885 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.827863 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:05:50.831840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:50.831821 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d05eb5-546bf789f8-cctdv"] Sep 16 05:05:51.121949 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:51.121921 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0a5d26-e818-428b-9837-4ac87536de07" path="/var/lib/kubelet/pods/1c0a5d26-e818-428b-9837-4ac87536de07/volumes" Sep 16 05:05:51.122329 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:05:51.122316 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="603da56e-c461-4027-90ea-4434357abee9" path="/var/lib/kubelet/pods/603da56e-c461-4027-90ea-4434357abee9/volumes" Sep 16 05:06:02.673663 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:02.673606 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" podUID="782850a2-6484-4812-92e7-3cd82c649ca8" containerName="main" containerID="cri-o://62aa4ca36152470300641c9feb770bd121208c2e0de35207e0a4f6032bac6a09" gracePeriod=45 Sep 16 05:06:02.858095 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:02.858066 2560 generic.go:356] "Generic (PLEG): container finished" podID="782850a2-6484-4812-92e7-3cd82c649ca8" containerID="62aa4ca36152470300641c9feb770bd121208c2e0de35207e0a4f6032bac6a09" exitCode=0 Sep 16 05:06:02.858219 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:02.858142 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" event={"ID":"782850a2-6484-4812-92e7-3cd82c649ca8","Type":"ContainerDied","Data":"62aa4ca36152470300641c9feb770bd121208c2e0de35207e0a4f6032bac6a09"} Sep 16 05:06:02.930682 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:02.930625 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:06:03.000687 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000661 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000697 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000722 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fkzs\" (UniqueName: \"kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000801 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000828 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000886 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home\") pod \"782850a2-6484-4812-92e7-3cd82c649ca8\" (UID: \"782850a2-6484-4812-92e7-3cd82c649ca8\") " Sep 16 05:06:03.000940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000919 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache" (OuterVolumeSpecName: "model-cache") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:03.001101 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.000943 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:03.001229 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.001206 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.001301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.001228 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home" (OuterVolumeSpecName: "home") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:03.001301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.001233 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.002940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.002916 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs" (OuterVolumeSpecName: "kube-api-access-4fkzs") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "kube-api-access-4fkzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:06:03.002940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.002927 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:06:03.003064 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.002977 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm" (OuterVolumeSpecName: "dshm") pod "782850a2-6484-4812-92e7-3cd82c649ca8" (UID: "782850a2-6484-4812-92e7-3cd82c649ca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:03.102520 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.102497 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkzs\" (UniqueName: \"kubernetes.io/projected/782850a2-6484-4812-92e7-3cd82c649ca8-kube-api-access-4fkzs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.102520 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.102519 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.102684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.102530 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/782850a2-6484-4812-92e7-3cd82c649ca8-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.102684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.102538 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/782850a2-6484-4812-92e7-3cd82c649ca8-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:03.865426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.865396 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" Sep 16 05:06:03.865929 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.865424 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn" event={"ID":"782850a2-6484-4812-92e7-3cd82c649ca8","Type":"ContainerDied","Data":"9d30acc5a2e7fb0a0e93460718d0b2b1a9e36174cf67298f59f9af8a84abe62f"} Sep 16 05:06:03.865929 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.865589 2560 scope.go:117] "RemoveContainer" containerID="62aa4ca36152470300641c9feb770bd121208c2e0de35207e0a4f6032bac6a09" Sep 16 05:06:03.883199 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.883170 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:06:03.889759 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:03.889736 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-d05eb5/autores-model-d05eb5-kserve-76c88944d7-2hvrn"] Sep 16 05:06:05.121485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:05.121449 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782850a2-6484-4812-92e7-3cd82c649ca8" path="/var/lib/kubelet/pods/782850a2-6484-4812-92e7-3cd82c649ca8/volumes" Sep 16 05:06:14.463146 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463111 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:06:14.463762 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463742 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0a5d26-e818-428b-9837-4ac87536de07" containerName="payload-pre-processing" Sep 16 05:06:14.463848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463765 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0a5d26-e818-428b-9837-4ac87536de07" containerName="payload-pre-processing" Sep 16 05:06:14.463848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463815 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782850a2-6484-4812-92e7-3cd82c649ca8" containerName="main" Sep 16 05:06:14.463848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463825 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="782850a2-6484-4812-92e7-3cd82c649ca8" containerName="main" Sep 16 05:06:14.463848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463841 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603da56e-c461-4027-90ea-4434357abee9" containerName="maas-api" Sep 16 05:06:14.463848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463849 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="603da56e-c461-4027-90ea-4434357abee9" containerName="maas-api" Sep 16 05:06:14.464094 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463971 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="782850a2-6484-4812-92e7-3cd82c649ca8" containerName="main" Sep 16 05:06:14.464094 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463986 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0a5d26-e818-428b-9837-4ac87536de07" containerName="payload-pre-processing" Sep 16 05:06:14.464094 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.463997 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="603da56e-c461-4027-90ea-4434357abee9" containerName="maas-api" Sep 16 05:06:14.468518 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.468498 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.471199 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.471142 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"openshift-service-ca.crt\"" Sep 16 05:06:14.472627 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.472414 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"kube-root-ca.crt\"" Sep 16 05:06:14.472747 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.472686 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"premium-w4-12b6c6-kserve-self-signed-certs\"" Sep 16 05:06:14.472887 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.472870 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"default-dockercfg-j9d82\"" Sep 16 05:06:14.476741 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.476720 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:06:14.512867 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.512840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.512976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.512878 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.512976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.512903 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.512976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.512953 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.513137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.512994 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.513137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.513034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614078 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614170 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614194 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614220 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614478 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614512 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.614642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.614601 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.616454 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.616431 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.616640 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.616622 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.624161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.624127 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v\") pod \"premium-w4-12b6c6-kserve-5d4b89cd86-99cnw\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:14.778832 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:14.778758 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:15.053359 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:15.053326 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:06:15.053834 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:06:15.053811 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67797014_6b4c_4b9b_8b98_0fa55a184c62.slice/crio-192e37298c39bf46948560e0008c3709cb574a446a9c035a2872512b8ece0df0 WatchSource:0}: Error finding container 192e37298c39bf46948560e0008c3709cb574a446a9c035a2872512b8ece0df0: Status 404 returned error can't find the container with id 192e37298c39bf46948560e0008c3709cb574a446a9c035a2872512b8ece0df0 Sep 16 05:06:15.940046 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:15.940007 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" event={"ID":"67797014-6b4c-4b9b-8b98-0fa55a184c62","Type":"ContainerStarted","Data":"68e5d6fad76292e123d320a9aee8d70bb8dbcf9f33254c23b5ff0fdb655dddb7"} Sep 16 05:06:15.940429 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:15.940053 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" event={"ID":"67797014-6b4c-4b9b-8b98-0fa55a184c62","Type":"ContainerStarted","Data":"192e37298c39bf46948560e0008c3709cb574a446a9c035a2872512b8ece0df0"} Sep 16 05:06:15.959070 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:15.959024 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" podStartSLOduration=1.959008817 podStartE2EDuration="1.959008817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:06:15.95583393 +0000 UTC m=+1041.469385478" watchObservedRunningTime="2026-09-16 05:06:15.959008817 +0000 UTC m=+1041.472560384" Sep 16 05:06:24.779884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:24.779851 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:24.780298 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:24.779902 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:24.792543 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:24.792515 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:25.013249 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:25.013220 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:06:32.582677 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:32.582639 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:06:32.583040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:32.582870 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" containerID="cri-o://f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9" gracePeriod=30 Sep 16 05:06:33.310253 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.310223 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:06:33.313954 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.313933 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.316814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.316793 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-672f57-openshift-default-dockercfg-l28xq\"" Sep 16 05:06:33.324461 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.324435 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:06:33.402061 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.401983 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402061 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402100 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402126 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402181 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402428 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402321 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402428 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.402428 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.402370 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bn9j\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.503909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.503873 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.503909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.503914 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.503950 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.503987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504056 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bn9j\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504179 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504457 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504689 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504582 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504689 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504643 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.504932 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.504914 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.506516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.506496 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.506620 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.506604 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.512426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.512404 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.512529 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.512478 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bn9j\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j\") pod \"e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.627149 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.627119 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:33.908310 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.908127 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:06:33.909330 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:06:33.909304 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381faf60_f3ae_4266_81ec_e44559e499bd.slice/crio-db1f5fd29456bb8b07808bdfc2959959ff42c9c0a0c439615a25b99dd5e369e0 WatchSource:0}: Error finding container db1f5fd29456bb8b07808bdfc2959959ff42c9c0a0c439615a25b99dd5e369e0: Status 404 returned error can't find the container with id db1f5fd29456bb8b07808bdfc2959959ff42c9c0a0c439615a25b99dd5e369e0 Sep 16 05:06:33.911749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.911717 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:06:33.911815 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.911787 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:06:33.911855 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:33.911817 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:06:34.057447 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:34.057412 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerStarted","Data":"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816"} Sep 16 05:06:34.057704 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:34.057464 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerStarted","Data":"db1f5fd29456bb8b07808bdfc2959959ff42c9c0a0c439615a25b99dd5e369e0"} Sep 16 05:06:34.084911 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:34.084849 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podStartSLOduration=1.084832628 podStartE2EDuration="1.084832628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:06:34.083596651 +0000 UTC m=+1059.597148185" watchObservedRunningTime="2026-09-16 05:06:34.084832628 +0000 UTC m=+1059.598384162" Sep 16 05:06:34.627943 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:34.627913 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:34.632796 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:34.632771 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:35.064902 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:35.064822 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:35.065938 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:35.065911 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:06:37.844665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.844588 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:06:37.850943 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.850922 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851070 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.850961 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851070 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.850991 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851070 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851033 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851070 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851057 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851270 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851117 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfsmc\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851270 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851144 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851270 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851152 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:37.851270 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851257 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851494 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851303 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token\") pod \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\" (UID: \"60aa236f-12ed-40fc-92fe-d67bc473f3fc\") " Sep 16 05:06:37.851494 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851318 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:37.851494 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851426 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:37.851656 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851538 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:06:37.851656 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851566 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:37.851789 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851770 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.851847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851795 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.851847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851810 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.851847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851822 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.851847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.851835 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.853269 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.853237 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:06:37.853364 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.853267 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:06:37.853423 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.853391 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc" (OuterVolumeSpecName: "kube-api-access-kfsmc") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "kube-api-access-kfsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:06:37.853464 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.853450 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "60aa236f-12ed-40fc-92fe-d67bc473f3fc" (UID: "60aa236f-12ed-40fc-92fe-d67bc473f3fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:06:37.952389 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.952346 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.952389 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.952382 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.952389 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.952391 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60aa236f-12ed-40fc-92fe-d67bc473f3fc-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:37.952709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:37.952401 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfsmc\" (UniqueName: \"kubernetes.io/projected/60aa236f-12ed-40fc-92fe-d67bc473f3fc-kube-api-access-kfsmc\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:06:38.084989 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.084954 2560 generic.go:356] "Generic (PLEG): container finished" podID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerID="f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9" exitCode=0 Sep 16 05:06:38.085145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.085012 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" Sep 16 05:06:38.085145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.085038 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerDied","Data":"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9"} Sep 16 05:06:38.085145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.085116 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j" event={"ID":"60aa236f-12ed-40fc-92fe-d67bc473f3fc","Type":"ContainerDied","Data":"e99eb6212f4cace9a0a23cbe7a2b1c22835188cc6bcce5beadbcc278b7fc6c4f"} Sep 16 05:06:38.085145 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.085140 2560 scope.go:117] "RemoveContainer" containerID="f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9" Sep 16 05:06:38.094452 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.094436 2560 scope.go:117] "RemoveContainer" containerID="f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c" Sep 16 05:06:38.102775 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.102759 2560 scope.go:117] "RemoveContainer" containerID="f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9" Sep 16 05:06:38.103007 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:06:38.102989 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9\": container with ID starting with f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9 not found: ID does not exist" containerID="f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9" Sep 16 05:06:38.103080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.103013 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9"} err="failed to get container status \"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9\": rpc error: code = NotFound desc = could not find container \"f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9\": container with ID starting with f3c8aa7fb1f74ad4416a698af83bc59ca18813242866eb936a7c87633a5ab7d9 not found: ID does not exist" Sep 16 05:06:38.103080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.103027 2560 scope.go:117] "RemoveContainer" containerID="f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c" Sep 16 05:06:38.103276 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:06:38.103252 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c\": container with ID starting with f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c not found: ID does not exist" containerID="f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c" Sep 16 05:06:38.103420 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.103282 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c"} err="failed to get container status \"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c\": rpc error: code = NotFound desc = could not find container \"f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c\": container with ID starting with f862964e6c7e02040a2458e0667f6bcd3ea3c8907bde00d1e8d2a2618cc94d0c not found: ID does not exist" Sep 16 05:06:38.107480 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.107460 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:06:38.113639 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:38.113617 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d05eb5-openshift-default-7875475df6-n682j"] Sep 16 05:06:39.120949 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:06:39.120917 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" path="/var/lib/kubelet/pods/60aa236f-12ed-40fc-92fe-d67bc473f3fc/volumes" Sep 16 05:07:01.235096 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.235005 2560 generic.go:356] "Generic (PLEG): container finished" podID="381faf60-f3ae-4266-81ec-e44559e499bd" containerID="0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816" exitCode=0 Sep 16 05:07:01.235642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.235083 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerDied","Data":"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816"} Sep 16 05:07:01.235718 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.235675 2560 scope.go:117] "RemoveContainer" containerID="0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816" Sep 16 05:07:01.243746 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.243702 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:01.243858 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.243797 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:01.243901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:01.243836 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:02.243893 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:02.243856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerStarted","Data":"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4"} Sep 16 05:07:02.627959 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:02.627925 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:07:02.629054 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:02.629026 2560 patch_prober.go:28] interesting pod/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 16 05:07:02.629211 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:02.629085 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 16 05:07:03.628519 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:03.628478 2560 patch_prober.go:28] interesting pod/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 16 05:07:03.628953 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:03.628575 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 16 05:07:04.097117 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097081 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:07:04.097505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097490 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.097505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097504 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.097681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097525 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.097681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097531 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.097681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097623 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.097681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.097631 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="60aa236f-12ed-40fc-92fe-d67bc473f3fc" containerName="istio-proxy" Sep 16 05:07:04.100971 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.100950 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.104202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.104174 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a28799ba-openshift-default-dockercfg-tp5rl\"" Sep 16 05:07:04.114519 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.114496 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:07:04.205899 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.205867 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.205930 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.205968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.205993 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.206065 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5b5\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206275 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.206087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206275 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.206103 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206275 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.206148 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.206275 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.206180 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307626 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307589 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5b5\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307626 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307649 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307671 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307714 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307749 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.307878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307863 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307934 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.307983 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.308053 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.308079 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.308137 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308438 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.308420 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.308681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.308658 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.310150 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.310125 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.310327 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.310308 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.319996 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.319972 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.321037 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.321017 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5b5\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5\") pod \"e2e-mt-a28799ba-openshift-default-847954577b-skk4q\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.413136 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.413050 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:04.628460 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.628420 2560 patch_prober.go:28] interesting pod/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 16 05:07:04.628635 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.628497 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 16 05:07:04.793626 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:07:04.793595 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed7b888_b212_4cc2_96d8_341bf052fb69.slice/crio-faca803ce30f3df7462f05fe41e9a968824027c6b235b98b05e239716af110d9 WatchSource:0}: Error finding container faca803ce30f3df7462f05fe41e9a968824027c6b235b98b05e239716af110d9: Status 404 returned error can't find the container with id faca803ce30f3df7462f05fe41e9a968824027c6b235b98b05e239716af110d9 Sep 16 05:07:04.796027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.795991 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:04.796115 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.796067 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:04.796157 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.796109 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:04.806614 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:04.806584 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:07:05.065720 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.065623 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:07:05.271576 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.271156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerStarted","Data":"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4"} Sep 16 05:07:05.271576 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.271225 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerStarted","Data":"faca803ce30f3df7462f05fe41e9a968824027c6b235b98b05e239716af110d9"} Sep 16 05:07:05.294838 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.294788 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podStartSLOduration=1.294771804 podStartE2EDuration="1.294771804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:07:05.292082083 +0000 UTC m=+1090.805633629" watchObservedRunningTime="2026-09-16 05:07:05.294771804 +0000 UTC m=+1090.808323338" Sep 16 05:07:05.414445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.414001 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:05.420003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.419981 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:05.628411 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.628374 2560 patch_prober.go:28] interesting pod/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 16 05:07:05.628612 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:05.628433 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 16 05:07:06.278176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:06.278144 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:06.279225 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:06.279208 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:06.628334 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:06.628297 2560 patch_prober.go:28] interesting pod/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Sep 16 05:07:06.628521 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:06.628351 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Sep 16 05:07:07.632132 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:07.632102 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:07:07.633143 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:07.633121 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:07:34.456061 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.455958 2560 generic.go:356] "Generic (PLEG): container finished" podID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerID="b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4" exitCode=0 Sep 16 05:07:34.456061 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.456033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerDied","Data":"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4"} Sep 16 05:07:34.456658 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.456420 2560 scope.go:117] "RemoveContainer" containerID="b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4" Sep 16 05:07:34.457542 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.457510 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:34.457656 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.457621 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:34.457733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:34.457713 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:07:35.480562 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:35.480509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerStarted","Data":"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059"} Sep 16 05:07:36.279207 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:36.279164 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:36.413910 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:36.413874 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:36.415204 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:36.415174 2560 patch_prober.go:28] interesting pod/e2e-mt-a28799ba-openshift-default-847954577b-skk4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 05:07:36.415328 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:36.415229 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 05:07:37.413622 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:37.413586 2560 patch_prober.go:28] interesting pod/e2e-mt-a28799ba-openshift-default-847954577b-skk4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 05:07:37.414038 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:37.413657 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 05:07:38.413877 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:38.413841 2560 patch_prober.go:28] interesting pod/e2e-mt-a28799ba-openshift-default-847954577b-skk4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 05:07:38.414277 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:38.413933 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 05:07:39.414294 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.414261 2560 patch_prober.go:28] interesting pod/e2e-mt-a28799ba-openshift-default-847954577b-skk4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 05:07:39.414807 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.414349 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 05:07:39.947179 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.947143 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:07:39.966839 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.966808 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:39.980619 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.980584 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:07:39.981234 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.981210 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-672f57\"/\"kube-root-ca.crt\"" Sep 16 05:07:39.981688 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.981670 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-672f57\"/\"openshift-service-ca.crt\"" Sep 16 05:07:39.982018 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.982001 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-672f57\"/\"explicit-model-672f57-kserve-self-signed-certs\"" Sep 16 05:07:39.982362 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:39.982346 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-672f57\"/\"default-dockercfg-wqqr7\"" Sep 16 05:07:40.075195 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n225j\" (UniqueName: \"kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.075388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075217 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.075388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.075511 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075491 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.075623 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.075666 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.075633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.176417 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.176377 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.176700 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.176679 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.176830 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.176814 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n225j\" (UniqueName: \"kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.176947 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.176857 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.177095 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.177081 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.177321 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.177304 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.177754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.177737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.178298 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.177678 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.178473 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.177170 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.179793 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.179766 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.180617 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.180598 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.184503 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.184464 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n225j\" (UniqueName: \"kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j\") pod \"explicit-model-672f57-kserve-7885dbd55f-65fzq\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.291710 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.291612 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:40.413819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.413779 2560 patch_prober.go:28] interesting pod/e2e-mt-a28799ba-openshift-default-847954577b-skk4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 16 05:07:40.413999 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.413839 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 16 05:07:40.573657 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:40.573625 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:07:40.575427 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:07:40.575390 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19db8134_c736_4179_8d89_3ee76b1ae96a.slice/crio-8d03752ae8323078a30943a4414ee80fc6a69b8b0cee9c4dc572e6f96e484862 WatchSource:0}: Error finding container 8d03752ae8323078a30943a4414ee80fc6a69b8b0cee9c4dc572e6f96e484862: Status 404 returned error can't find the container with id 8d03752ae8323078a30943a4414ee80fc6a69b8b0cee9c4dc572e6f96e484862 Sep 16 05:07:41.417522 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:41.417487 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:41.418585 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:41.418561 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:07:41.524113 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:41.524069 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" event={"ID":"19db8134-c736-4179-8d89-3ee76b1ae96a","Type":"ContainerStarted","Data":"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056"} Sep 16 05:07:41.524276 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:41.524127 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" event={"ID":"19db8134-c736-4179-8d89-3ee76b1ae96a","Type":"ContainerStarted","Data":"8d03752ae8323078a30943a4414ee80fc6a69b8b0cee9c4dc572e6f96e484862"} Sep 16 05:07:41.545467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:41.545411 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" podStartSLOduration=2.545391999 podStartE2EDuration="2.545391999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:07:41.543758484 +0000 UTC m=+1127.057310018" watchObservedRunningTime="2026-09-16 05:07:41.545391999 +0000 UTC m=+1127.058943532" Sep 16 05:07:48.651715 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.651671 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:07:48.655887 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.655865 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.658125 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.658106 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-12b6c6\"/\"e2e-conflict-a-2a88c17b-kserve-self-signed-certs\"" Sep 16 05:07:48.665831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.665811 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:07:48.765814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.765780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.765924 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.765852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzvw\" (UniqueName: \"kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.765924 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.765892 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.766014 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.765979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.766014 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.766006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.766081 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.766061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.866871 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.866842 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867043 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.866907 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867043 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.866928 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867043 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.866966 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867208 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.867043 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867208 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.867085 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzvw\" (UniqueName: \"kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.867246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.867299 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.867435 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.867376 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.869425 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.869406 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.869560 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.869531 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.874504 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.874484 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzvw\" (UniqueName: \"kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw\") pod \"e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:48.967595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:48.967500 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:49.246703 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:49.246676 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:07:49.247385 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:07:49.247354 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fd23d95_8741_453d_bd73_b96d525623ba.slice/crio-715b09c3ca266c11d25884ab1edb8d6a058c46ff661c7e56271d36e944774eba WatchSource:0}: Error finding container 715b09c3ca266c11d25884ab1edb8d6a058c46ff661c7e56271d36e944774eba: Status 404 returned error can't find the container with id 715b09c3ca266c11d25884ab1edb8d6a058c46ff661c7e56271d36e944774eba Sep 16 05:07:49.580212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:49.580180 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" event={"ID":"2fd23d95-8741-453d-bd73-b96d525623ba","Type":"ContainerStarted","Data":"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf"} Sep 16 05:07:49.580366 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:49.580222 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" event={"ID":"2fd23d95-8741-453d-bd73-b96d525623ba","Type":"ContainerStarted","Data":"715b09c3ca266c11d25884ab1edb8d6a058c46ff661c7e56271d36e944774eba"} Sep 16 05:07:49.599404 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:49.599349 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" podStartSLOduration=1.59933449 podStartE2EDuration="1.59933449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:07:49.597609097 +0000 UTC m=+1135.111160631" watchObservedRunningTime="2026-09-16 05:07:49.59933449 +0000 UTC m=+1135.112886023" Sep 16 05:07:50.292328 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:50.292294 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:50.292780 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:50.292345 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:50.305117 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:50.305090 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:50.597560 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:50.597514 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:07:51.541350 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:51.541313 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:07:51.697052 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:51.697020 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:51.697239 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:51.697094 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:07:52.197075485 +0000 UTC m=+1137.710626996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:52.204076 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:52.204044 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:52.204252 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:52.204117 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:07:53.204099957 +0000 UTC m=+1138.717651474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:53.215592 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:53.215540 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:53.215970 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:53.215632 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:07:55.215617094 +0000 UTC m=+1140.729168607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:55.236821 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:55.236774 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:55.237210 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:55.236865 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:07:59.236851965 +0000 UTC m=+1144.750403481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:58.967754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:58.967702 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:58.967754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:58.967771 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:58.980570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:58.980527 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:07:59.280829 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:59.280745 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:59.280829 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:07:59.280822 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:08:07.280804586 +0000 UTC m=+1152.794356112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:07:59.660429 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:07:59.660399 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:08:04.933896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:04.933862 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:08:04.934497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:04.934157 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" containerID="cri-o://a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059" gracePeriod=30 Sep 16 05:08:07.374266 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:07.374228 2560 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-self-signed-certs: secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:08:07.374676 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:07.374311 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs podName:19db8134-c736-4179-8d89-3ee76b1ae96a nodeName:}" failed. No retries permitted until 2026-09-16 05:08:23.374296102 +0000 UTC m=+1168.887847613 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs") pod "explicit-model-672f57-kserve-7885dbd55f-65fzq" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a") : secret "explicit-model-672f57-kserve-self-signed-certs" not found Sep 16 05:08:07.618067 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.618030 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" podUID="19db8134-c736-4179-8d89-3ee76b1ae96a" containerName="main" containerID="cri-o://175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056" gracePeriod=45 Sep 16 05:08:07.878532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.878508 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:08:07.979702 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979622 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.979702 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979667 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n225j\" (UniqueName: \"kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.979702 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979696 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.979979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.979979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979842 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.979979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979889 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir\") pod \"19db8134-c736-4179-8d89-3ee76b1ae96a\" (UID: \"19db8134-c736-4179-8d89-3ee76b1ae96a\") " Sep 16 05:08:07.980140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.979881 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache" (OuterVolumeSpecName: "model-cache") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:07.980200 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.980174 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:07.980273 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.980239 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home" (OuterVolumeSpecName: "home") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:07.980344 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.980317 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:07.980344 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.980333 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:07.982150 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.982120 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm" (OuterVolumeSpecName: "dshm") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:07.982328 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.982305 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:08:07.982383 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:07.982328 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j" (OuterVolumeSpecName: "kube-api-access-n225j") pod "19db8134-c736-4179-8d89-3ee76b1ae96a" (UID: "19db8134-c736-4179-8d89-3ee76b1ae96a"). InnerVolumeSpecName "kube-api-access-n225j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:08:08.081331 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.081300 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n225j\" (UniqueName: \"kubernetes.io/projected/19db8134-c736-4179-8d89-3ee76b1ae96a-kube-api-access-n225j\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:08.081331 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.081329 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19db8134-c736-4179-8d89-3ee76b1ae96a-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:08.081491 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.081342 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:08.081491 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.081352 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19db8134-c736-4179-8d89-3ee76b1ae96a-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:08.702988 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.702947 2560 generic.go:356] "Generic (PLEG): container finished" podID="19db8134-c736-4179-8d89-3ee76b1ae96a" containerID="175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056" exitCode=0 Sep 16 05:08:08.702988 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.702991 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" Sep 16 05:08:08.703504 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.703010 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" event={"ID":"19db8134-c736-4179-8d89-3ee76b1ae96a","Type":"ContainerDied","Data":"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056"} Sep 16 05:08:08.703504 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.703050 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq" event={"ID":"19db8134-c736-4179-8d89-3ee76b1ae96a","Type":"ContainerDied","Data":"8d03752ae8323078a30943a4414ee80fc6a69b8b0cee9c4dc572e6f96e484862"} Sep 16 05:08:08.703504 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.703071 2560 scope.go:117] "RemoveContainer" containerID="175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056" Sep 16 05:08:08.713685 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.713664 2560 scope.go:117] "RemoveContainer" containerID="175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056" Sep 16 05:08:08.713958 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:08.713936 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056\": container with ID starting with 175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056 not found: ID does not exist" containerID="175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056" Sep 16 05:08:08.714017 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.713968 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056"} err="failed to get container status \"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056\": rpc error: code = NotFound desc = could not find container \"175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056\": container with ID starting with 175aad1753dc20438802567feb20e778c1bb0e149d1558c872e3dcbdc930a056 not found: ID does not exist" Sep 16 05:08:08.725485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.725460 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:08:08.732042 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:08.732020 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-672f57/explicit-model-672f57-kserve-7885dbd55f-65fzq"] Sep 16 05:08:09.121763 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:09.121732 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19db8134-c736-4179-8d89-3ee76b1ae96a" path="/var/lib/kubelet/pods/19db8134-c736-4179-8d89-3ee76b1ae96a/volumes" Sep 16 05:08:10.020985 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.020952 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:08:10.021354 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.021209 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" containerID="cri-o://40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4" gracePeriod=30 Sep 16 05:08:10.195590 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.195518 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:08:10.300881 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.300854 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301042 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.300895 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301042 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.300921 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301170 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301040 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301170 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301069 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301170 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301099 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301170 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301137 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301205 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s5b5\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5\") pod \"9ed7b888-b212-4cc2-96d8-341bf052fb69\" (UID: \"9ed7b888-b212-4cc2-96d8-341bf052fb69\") " Sep 16 05:08:10.301388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301336 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:10.301570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301416 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:10.301570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301449 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:08:10.301570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301455 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:10.301570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301534 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:10.301795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301740 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.301795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301759 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.301795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301775 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ed7b888-b212-4cc2-96d8-341bf052fb69-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.301795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301789 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.302013 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.301802 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.303205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.303183 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:10.303205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.303185 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:08:10.303339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.303213 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:08:10.303339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.303308 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5" (OuterVolumeSpecName: "kube-api-access-6s5b5") pod "9ed7b888-b212-4cc2-96d8-341bf052fb69" (UID: "9ed7b888-b212-4cc2-96d8-341bf052fb69"). InnerVolumeSpecName "kube-api-access-6s5b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:08:10.402871 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.402848 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s5b5\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-kube-api-access-6s5b5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.402871 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.402872 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.403005 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.402882 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.403005 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.402890 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ed7b888-b212-4cc2-96d8-341bf052fb69-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:10.718537 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.718503 2560 generic.go:356] "Generic (PLEG): container finished" podID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerID="a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059" exitCode=0 Sep 16 05:08:10.718709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.718572 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" Sep 16 05:08:10.718709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.718598 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerDied","Data":"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059"} Sep 16 05:08:10.718709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.718660 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q" event={"ID":"9ed7b888-b212-4cc2-96d8-341bf052fb69","Type":"ContainerDied","Data":"faca803ce30f3df7462f05fe41e9a968824027c6b235b98b05e239716af110d9"} Sep 16 05:08:10.718709 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.718691 2560 scope.go:117] "RemoveContainer" containerID="a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059" Sep 16 05:08:10.731841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.730033 2560 scope.go:117] "RemoveContainer" containerID="b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4" Sep 16 05:08:10.741102 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.741083 2560 scope.go:117] "RemoveContainer" containerID="a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059" Sep 16 05:08:10.741403 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:10.741381 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059\": container with ID starting with a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059 not found: ID does not exist" containerID="a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059" Sep 16 05:08:10.741488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.741410 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059"} err="failed to get container status \"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059\": rpc error: code = NotFound desc = could not find container \"a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059\": container with ID starting with a3382352817af87cf932828ce002ad06eb56fb1d0e2cea313268f277bbe21059 not found: ID does not exist" Sep 16 05:08:10.741488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.741427 2560 scope.go:117] "RemoveContainer" containerID="b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4" Sep 16 05:08:10.741946 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:10.741815 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4\": container with ID starting with b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4 not found: ID does not exist" containerID="b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4" Sep 16 05:08:10.742057 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.741982 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4"} err="failed to get container status \"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4\": rpc error: code = NotFound desc = could not find container \"b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4\": container with ID starting with b036458ddddb5509936dffa8b1e10fdd682e2fe87c1be171c024489ad65f69a4 not found: ID does not exist" Sep 16 05:08:10.747768 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.747745 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:08:10.755310 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:10.755285 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a28799ba-openshift-default-847954577b-skk4q"] Sep 16 05:08:11.129996 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:11.129963 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" path="/var/lib/kubelet/pods/9ed7b888-b212-4cc2-96d8-341bf052fb69/volumes" Sep 16 05:08:15.283992 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.283923 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:08:15.452618 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452582 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.452803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452625 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.452803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452661 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.452803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452685 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.452803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452713 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.452803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452749 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452846 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bn9j\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452877 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452875 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.452983 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453024 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:15.453080 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453047 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy\") pod \"381faf60-f3ae-4266-81ec-e44559e499bd\" (UID: \"381faf60-f3ae-4266-81ec-e44559e499bd\") " Sep 16 05:08:15.453379 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453140 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:15.453379 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453161 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:08:15.453490 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453394 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.453490 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453416 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/381faf60-f3ae-4266-81ec-e44559e499bd-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.453490 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453430 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.453490 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453442 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.453490 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.453454 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.455048 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.455022 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:08:15.455160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.455079 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:08:15.455160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.455085 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j" (OuterVolumeSpecName: "kube-api-access-4bn9j") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "kube-api-access-4bn9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:08:15.455160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.455135 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "381faf60-f3ae-4266-81ec-e44559e499bd" (UID: "381faf60-f3ae-4266-81ec-e44559e499bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:08:15.554962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.554888 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/381faf60-f3ae-4266-81ec-e44559e499bd-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.554962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.554913 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.554962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.554921 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/381faf60-f3ae-4266-81ec-e44559e499bd-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.554962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.554930 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bn9j\" (UniqueName: \"kubernetes.io/projected/381faf60-f3ae-4266-81ec-e44559e499bd-kube-api-access-4bn9j\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:08:15.751224 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.751190 2560 generic.go:356] "Generic (PLEG): container finished" podID="381faf60-f3ae-4266-81ec-e44559e499bd" containerID="40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4" exitCode=0 Sep 16 05:08:15.751409 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.751233 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" Sep 16 05:08:15.751409 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.751244 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerDied","Data":"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4"} Sep 16 05:08:15.751409 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.751287 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr" event={"ID":"381faf60-f3ae-4266-81ec-e44559e499bd","Type":"ContainerDied","Data":"db1f5fd29456bb8b07808bdfc2959959ff42c9c0a0c439615a25b99dd5e369e0"} Sep 16 05:08:15.751409 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.751308 2560 scope.go:117] "RemoveContainer" containerID="40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4" Sep 16 05:08:15.761469 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.761449 2560 scope.go:117] "RemoveContainer" containerID="0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816" Sep 16 05:08:15.770334 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.770295 2560 scope.go:117] "RemoveContainer" containerID="40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4" Sep 16 05:08:15.770670 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:15.770650 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4\": container with ID starting with 40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4 not found: ID does not exist" containerID="40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4" Sep 16 05:08:15.770748 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.770677 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4"} err="failed to get container status \"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4\": rpc error: code = NotFound desc = could not find container \"40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4\": container with ID starting with 40dc2e42a7e29627ce3407238e1acc056884888d203baa049410850c682e58e4 not found: ID does not exist" Sep 16 05:08:15.770748 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.770696 2560 scope.go:117] "RemoveContainer" containerID="0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816" Sep 16 05:08:15.770947 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:15.770933 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816\": container with ID starting with 0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816 not found: ID does not exist" containerID="0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816" Sep 16 05:08:15.770984 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.770949 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816"} err="failed to get container status \"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816\": rpc error: code = NotFound desc = could not find container \"0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816\": container with ID starting with 0fccb3be24b651b791fefa38370aeb3a9f17eab7e2e9ff27147e06bc10cd4816 not found: ID does not exist" Sep 16 05:08:15.773225 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.773197 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:08:15.776882 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:15.776862 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-672f57-openshift-default-9c6447bdd-jxpgr"] Sep 16 05:08:17.122440 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:17.122404 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" path="/var/lib/kubelet/pods/381faf60-f3ae-4266-81ec-e44559e499bd/volumes" Sep 16 05:08:18.922509 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.922478 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:08:18.922961 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.922946 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:18.923006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.922963 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:18.923006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.922975 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19db8134-c736-4179-8d89-3ee76b1ae96a" containerName="main" Sep 16 05:08:18.923006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.922981 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="19db8134-c736-4179-8d89-3ee76b1ae96a" containerName="main" Sep 16 05:08:18.923006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923001 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:18.923006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923007 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:18.923181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923014 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:18.923181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923019 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:18.923181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923090 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:18.923181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923099 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:18.923181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.923106 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="19db8134-c736-4179-8d89-3ee76b1ae96a" containerName="main" Sep 16 05:08:18.927713 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.927685 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:18.932191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.932169 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-ad06de-openshift-default-dockercfg-r69mt\"" Sep 16 05:08:18.943419 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:18.943396 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:08:19.089587 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089531 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.089721 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089627 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.089721 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089699 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.089844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089728 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.089844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089748 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.089844 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.090003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089846 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4qst\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.090003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089892 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.090003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.089950 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191239 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191157 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191239 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191249 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191276 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191302 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191330 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4qst\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191360 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191504 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191592 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191666 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.191811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191755 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.192019 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.191861 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.192137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.192119 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.193540 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.193521 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.193902 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.193883 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.205224 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.205195 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.205501 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.205483 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4qst\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst\") pod \"e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.239426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.239402 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:19.518478 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.518448 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:08:19.519216 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:08:19.519193 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8950fe98_8721_4f3d_9f26_d00b0e727938.slice/crio-9d91a32ac23ab6131e775f9460eaf19ca86acea8c6b937dd97a410f69dc0baab WatchSource:0}: Error finding container 9d91a32ac23ab6131e775f9460eaf19ca86acea8c6b937dd97a410f69dc0baab: Status 404 returned error can't find the container with id 9d91a32ac23ab6131e775f9460eaf19ca86acea8c6b937dd97a410f69dc0baab Sep 16 05:08:19.521309 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.521276 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:19.521390 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.521345 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:19.521390 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.521379 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:19.780536 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.780454 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerStarted","Data":"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564"} Sep 16 05:08:19.780536 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.780516 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerStarted","Data":"9d91a32ac23ab6131e775f9460eaf19ca86acea8c6b937dd97a410f69dc0baab"} Sep 16 05:08:19.804202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:19.804152 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podStartSLOduration=1.804136238 podStartE2EDuration="1.804136238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:08:19.801184715 +0000 UTC m=+1165.314736249" watchObservedRunningTime="2026-09-16 05:08:19.804136238 +0000 UTC m=+1165.317687772" Sep 16 05:08:20.240217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:20.240185 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:20.245077 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:20.245056 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:20.786968 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:20.786935 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:20.787918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:20.787896 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:24.823336 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.823304 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:08:24.823767 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.823753 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:24.823811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.823769 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:24.823860 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.823851 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed7b888-b212-4cc2-96d8-341bf052fb69" containerName="istio-proxy" Sep 16 05:08:24.823892 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.823865 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="381faf60-f3ae-4266-81ec-e44559e499bd" containerName="istio-proxy" Sep 16 05:08:24.827063 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.827042 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.829673 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.829652 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ad06de-serving-cert\"" Sep 16 05:08:24.839160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.839134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:08:24.846008 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.845979 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.846118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.846087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.846176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.846157 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djw9r\" (UniqueName: \"kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.846217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.846205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.947122 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.947086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djw9r\" (UniqueName: \"kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.947286 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.947136 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.947286 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.947241 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.947374 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.947306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.947418 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:24.947397 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-serving-cert: secret "maas-api-e2e-shared-a-w2-ad06de-serving-cert" not found Sep 16 05:08:24.947489 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:08:24.947478 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls podName:d6b3033b-3c96-4e9b-bb01-fe9284869f58 nodeName:}" failed. No retries permitted until 2026-09-16 05:08:25.447457073 +0000 UTC m=+1170.961008604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls") pod "maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" (UID: "d6b3033b-3c96-4e9b-bb01-fe9284869f58") : secret "maas-api-e2e-shared-a-w2-ad06de-serving-cert" not found Sep 16 05:08:24.947976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.947957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.949596 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.949579 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:24.955684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:24.955659 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djw9r\" (UniqueName: \"kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:25.452652 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:25.452612 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:25.454978 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:25.454957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:25.739811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:25.739731 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:26.013473 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.013442 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:08:26.015764 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:08:26.015735 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b3033b_3c96_4e9b_bb01_fe9284869f58.slice/crio-ecf395ef47f22d7115219c40425d0fd10252dc39c65bc6f10fcd9f8910dd791d WatchSource:0}: Error finding container ecf395ef47f22d7115219c40425d0fd10252dc39c65bc6f10fcd9f8910dd791d: Status 404 returned error can't find the container with id ecf395ef47f22d7115219c40425d0fd10252dc39c65bc6f10fcd9f8910dd791d Sep 16 05:08:26.017480 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.017462 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 05:08:26.835204 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.835169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" event={"ID":"d6b3033b-3c96-4e9b-bb01-fe9284869f58","Type":"ContainerStarted","Data":"0d7563df0cefa16bf9bcfed1c3d1c8619ed88c39c68fd2a482c6cf163d128cda"} Sep 16 05:08:26.835352 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.835248 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" event={"ID":"d6b3033b-3c96-4e9b-bb01-fe9284869f58","Type":"ContainerStarted","Data":"ecf395ef47f22d7115219c40425d0fd10252dc39c65bc6f10fcd9f8910dd791d"} Sep 16 05:08:26.835352 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.835299 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:26.855254 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:26.855201 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" podStartSLOduration=2.44507972 podStartE2EDuration="2.855186606s" totalImagesPullingTime="410.106886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:08:26.852733677 +0000 UTC m=+1172.366285211" watchObservedRunningTime="2026-09-16 05:08:26.855186606 +0000 UTC m=+1172.368738140" Sep 16 05:08:32.847829 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:32.847801 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:08:50.788561 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:50.788524 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:50.788931 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:50.788614 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:51.005989 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.005888 2560 generic.go:356] "Generic (PLEG): container finished" podID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerID="18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564" exitCode=0 Sep 16 05:08:51.005989 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.005941 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerDied","Data":"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564"} Sep 16 05:08:51.006370 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.006348 2560 scope.go:117] "RemoveContainer" containerID="18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564" Sep 16 05:08:51.007512 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.007476 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:51.007677 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.007583 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:51.007677 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:51.007659 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:08:52.014838 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:52.014804 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerStarted","Data":"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196"} Sep 16 05:08:52.239765 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:52.239733 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:52.239934 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:52.239902 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:52.239979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:52.239954 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:53.239840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:53.239801 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:53.240220 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:53.239860 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:54.240379 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:54.240346 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:54.240800 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:54.240402 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:55.046984 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.046952 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:08:55.048480 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.048459 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:08:55.069853 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.069825 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:08:55.076840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.076818 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:08:55.239899 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.239868 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:55.240035 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:55.239923 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:56.240213 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:56.240181 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Sep 16 05:08:56.240609 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:56.240234 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Sep 16 05:08:57.244231 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:57.244204 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:57.244609 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:57.244447 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:08:57.245032 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:08:57.245016 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:09:07.671871 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.671839 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:09:07.676891 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.676868 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.679883 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.679859 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-748b01\"/\"openshift-service-ca.crt\"" Sep 16 05:09:07.680009 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.679865 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-748b01\"/\"noten-model-748b01-kserve-self-signed-certs\"" Sep 16 05:09:07.680875 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.680852 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-748b01\"/\"kube-root-ca.crt\"" Sep 16 05:09:07.680953 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.680916 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-748b01\"/\"default-dockercfg-r8hgs\"" Sep 16 05:09:07.693886 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.693861 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:09:07.797113 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797083 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.797284 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.797284 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797210 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.797284 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.797396 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b7r7\" (UniqueName: \"kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.797396 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.797360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898316 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898284 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b7r7\" (UniqueName: \"kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898434 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898469 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898756 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898536 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898933 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898905 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.898933 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898920 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.899062 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.898954 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.900887 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.900859 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.901006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.900971 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.909754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.909732 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b7r7\" (UniqueName: \"kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7\") pod \"noten-model-748b01-kserve-7769bd86ff-29c2h\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:07.987455 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:07.987382 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:08.269381 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:08.269352 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:09:08.271638 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:08.271607 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cccd0c2_3749_46ef_893b_503b1097c7b2.slice/crio-2771b1d68ca2091cce99023bdb8984608d5126489c9a54a049c7f18aeb46d45f WatchSource:0}: Error finding container 2771b1d68ca2091cce99023bdb8984608d5126489c9a54a049c7f18aeb46d45f: Status 404 returned error can't find the container with id 2771b1d68ca2091cce99023bdb8984608d5126489c9a54a049c7f18aeb46d45f Sep 16 05:09:09.128055 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:09.128019 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" event={"ID":"6cccd0c2-3749-46ef-893b-503b1097c7b2","Type":"ContainerStarted","Data":"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de"} Sep 16 05:09:09.128055 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:09.128056 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" event={"ID":"6cccd0c2-3749-46ef-893b-503b1097c7b2","Type":"ContainerStarted","Data":"2771b1d68ca2091cce99023bdb8984608d5126489c9a54a049c7f18aeb46d45f"} Sep 16 05:09:09.150211 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:09.150167 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" podStartSLOduration=2.150154274 podStartE2EDuration="2.150154274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:09.147906145 +0000 UTC m=+1214.661457682" watchObservedRunningTime="2026-09-16 05:09:09.150154274 +0000 UTC m=+1214.663705818" Sep 16 05:09:15.509648 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.509613 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7844-04bfca"] Sep 16 05:09:15.513318 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.513301 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:15.516057 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.516033 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fr2c9\"" Sep 16 05:09:15.524384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.524363 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-04bfca"] Sep 16 05:09:15.681384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.681348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsdp\" (UniqueName: \"kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp\") pod \"test-curl-7844-04bfca\" (UID: \"f603ba13-e613-4b7b-bec9-c4726fbb469e\") " pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:15.782907 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.782818 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsdp\" (UniqueName: \"kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp\") pod \"test-curl-7844-04bfca\" (UID: \"f603ba13-e613-4b7b-bec9-c4726fbb469e\") " pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:15.791705 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.791674 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsdp\" (UniqueName: \"kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp\") pod \"test-curl-7844-04bfca\" (UID: \"f603ba13-e613-4b7b-bec9-c4726fbb469e\") " pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:15.823683 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:15.823645 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:16.106741 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:16.106708 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-04bfca"] Sep 16 05:09:16.107811 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:16.107782 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf603ba13_e613_4b7b_bec9_c4726fbb469e.slice/crio-ead71293f60ffb7de169faf34154007551093c078df7c06eb0a3064a723fd465 WatchSource:0}: Error finding container ead71293f60ffb7de169faf34154007551093c078df7c06eb0a3064a723fd465: Status 404 returned error can't find the container with id ead71293f60ffb7de169faf34154007551093c078df7c06eb0a3064a723fd465 Sep 16 05:09:16.172697 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:16.172652 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-04bfca" event={"ID":"f603ba13-e613-4b7b-bec9-c4726fbb469e","Type":"ContainerStarted","Data":"ead71293f60ffb7de169faf34154007551093c078df7c06eb0a3064a723fd465"} Sep 16 05:09:17.180730 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.180692 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-04bfca" event={"ID":"f603ba13-e613-4b7b-bec9-c4726fbb469e","Type":"ContainerStarted","Data":"8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271"} Sep 16 05:09:17.200302 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.200254 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7844-04bfca" podStartSLOduration=2.200237884 podStartE2EDuration="2.200237884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:17.196986903 +0000 UTC m=+1222.710538436" watchObservedRunningTime="2026-09-16 05:09:17.200237884 +0000 UTC m=+1222.713789450" Sep 16 05:09:17.846285 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.846235 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7844-04bfca"] Sep 16 05:09:17.849632 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.849607 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7844-04bfca"] Sep 16 05:09:17.961869 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.961777 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7844-d79e77"] Sep 16 05:09:17.962268 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.962247 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" containerName="test-curl-7844-04bfca" Sep 16 05:09:17.962268 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.962267 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" containerName="test-curl-7844-04bfca" Sep 16 05:09:17.962434 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.962372 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" containerName="test-curl-7844-04bfca" Sep 16 05:09:17.965586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.965567 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:17.972481 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.972458 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-d79e77"] Sep 16 05:09:17.987681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.987660 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:17.987804 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:17.987699 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:18.000819 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.000796 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:18.108755 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.108719 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x9vh\" (UniqueName: \"kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh\") pod \"test-curl-7844-d79e77\" (UID: \"35f49e10-92f8-4195-95c9-cdd962711b9d\") " pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:18.186557 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:18.186518 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7844-04bfca" container="test-curl-7844-04bfca" Sep 16 05:09:18.197641 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.197612 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:18.209708 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.209676 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x9vh\" (UniqueName: \"kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh\") pod \"test-curl-7844-d79e77\" (UID: \"35f49e10-92f8-4195-95c9-cdd962711b9d\") " pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:18.222182 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.222121 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x9vh\" (UniqueName: \"kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh\") pod \"test-curl-7844-d79e77\" (UID: \"35f49e10-92f8-4195-95c9-cdd962711b9d\") " pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:18.276752 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.276718 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:18.549909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:18.549880 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-d79e77"] Sep 16 05:09:18.550780 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:18.550749 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f49e10_92f8_4195_95c9_cdd962711b9d.slice/crio-e83f92cbed04d0bad26bcfa9c7b87c82c2a81da11931c1ee3995a91a1ce35350 WatchSource:0}: Error finding container e83f92cbed04d0bad26bcfa9c7b87c82c2a81da11931c1ee3995a91a1ce35350: Status 404 returned error can't find the container with id e83f92cbed04d0bad26bcfa9c7b87c82c2a81da11931c1ee3995a91a1ce35350 Sep 16 05:09:19.193577 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.193512 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-d79e77" event={"ID":"35f49e10-92f8-4195-95c9-cdd962711b9d","Type":"ContainerStarted","Data":"a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b"} Sep 16 05:09:19.194009 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.193594 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-d79e77" event={"ID":"35f49e10-92f8-4195-95c9-cdd962711b9d","Type":"ContainerStarted","Data":"e83f92cbed04d0bad26bcfa9c7b87c82c2a81da11931c1ee3995a91a1ce35350"} Sep 16 05:09:19.194009 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.193899 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7844-04bfca" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" containerName="test-curl-7844-04bfca" containerID="cri-o://8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271" gracePeriod=2 Sep 16 05:09:19.213029 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.212977 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7844-d79e77" podStartSLOduration=2.21296369 podStartE2EDuration="2.21296369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:19.210208353 +0000 UTC m=+1224.723759884" watchObservedRunningTime="2026-09-16 05:09:19.21296369 +0000 UTC m=+1224.726515250" Sep 16 05:09:19.857330 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.857294 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7844-d79e77"] Sep 16 05:09:19.860918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.860891 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7844-d79e77"] Sep 16 05:09:19.992151 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.992004 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7844-432d2d"] Sep 16 05:09:19.992921 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.992900 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" containerName="test-curl-7844-d79e77" Sep 16 05:09:19.993044 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.992925 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" containerName="test-curl-7844-d79e77" Sep 16 05:09:19.993146 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.993129 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" containerName="test-curl-7844-d79e77" Sep 16 05:09:19.996971 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:19.996947 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:20.006136 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.006074 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-432d2d"] Sep 16 05:09:20.130310 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.130277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpg5\" (UniqueName: \"kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5\") pod \"test-curl-7844-432d2d\" (UID: \"53fcd679-d16f-48ff-a7e2-4caf55cea127\") " pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:20.200375 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:20.200348 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7844-d79e77" container="test-curl-7844-d79e77" Sep 16 05:09:20.231320 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.231293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpg5\" (UniqueName: \"kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5\") pod \"test-curl-7844-432d2d\" (UID: \"53fcd679-d16f-48ff-a7e2-4caf55cea127\") " pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:20.241036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.241017 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpg5\" (UniqueName: \"kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5\") pod \"test-curl-7844-432d2d\" (UID: \"53fcd679-d16f-48ff-a7e2-4caf55cea127\") " pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:20.307017 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.306961 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:20.583115 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:20.583079 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-432d2d"] Sep 16 05:09:20.583850 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:20.583824 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53fcd679_d16f_48ff_a7e2_4caf55cea127.slice/crio-2ab57568c01293a57f5084a6ba1fab642ec4254fd1d00c52861a628d292a65b1 WatchSource:0}: Error finding container 2ab57568c01293a57f5084a6ba1fab642ec4254fd1d00c52861a628d292a65b1: Status 404 returned error can't find the container with id 2ab57568c01293a57f5084a6ba1fab642ec4254fd1d00c52861a628d292a65b1 Sep 16 05:09:21.211282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.211255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-432d2d" event={"ID":"53fcd679-d16f-48ff-a7e2-4caf55cea127","Type":"ContainerStarted","Data":"0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979"} Sep 16 05:09:21.211634 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.211302 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-432d2d" event={"ID":"53fcd679-d16f-48ff-a7e2-4caf55cea127","Type":"ContainerStarted","Data":"2ab57568c01293a57f5084a6ba1fab642ec4254fd1d00c52861a628d292a65b1"} Sep 16 05:09:21.211634 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.211311 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7844-d79e77" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" containerName="test-curl-7844-d79e77" containerID="cri-o://a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b" gracePeriod=2 Sep 16 05:09:21.229357 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.229296 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7844-432d2d" podStartSLOduration=2.2292755 podStartE2EDuration="2.2292755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:21.227724561 +0000 UTC m=+1226.741276111" watchObservedRunningTime="2026-09-16 05:09:21.2292755 +0000 UTC m=+1226.742827036" Sep 16 05:09:21.343506 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.343486 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:21.443849 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.443822 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gsdp\" (UniqueName: \"kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp\") pod \"f603ba13-e613-4b7b-bec9-c4726fbb469e\" (UID: \"f603ba13-e613-4b7b-bec9-c4726fbb469e\") " Sep 16 05:09:21.445842 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.445811 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp" (OuterVolumeSpecName: "kube-api-access-2gsdp") pod "f603ba13-e613-4b7b-bec9-c4726fbb469e" (UID: "f603ba13-e613-4b7b-bec9-c4726fbb469e"). InnerVolumeSpecName "kube-api-access-2gsdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:21.545108 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.545076 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gsdp\" (UniqueName: \"kubernetes.io/projected/f603ba13-e613-4b7b-bec9-c4726fbb469e-kube-api-access-2gsdp\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:21.842839 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.842810 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7844-432d2d"] Sep 16 05:09:21.847140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.847109 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7844-432d2d"] Sep 16 05:09:21.959657 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.959622 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7844-3923ac"] Sep 16 05:09:21.960076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.960063 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" containerName="test-curl-7844-432d2d" Sep 16 05:09:21.960120 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.960078 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" containerName="test-curl-7844-432d2d" Sep 16 05:09:21.960159 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.960151 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" containerName="test-curl-7844-432d2d" Sep 16 05:09:21.963338 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.963319 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:21.967787 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:21.967758 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-3923ac"] Sep 16 05:09:22.151160 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.151078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx82v\" (UniqueName: \"kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v\") pod \"test-curl-7844-3923ac\" (UID: \"d15e4abb-c401-435d-8863-318be2c9c2fa\") " pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:22.218967 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.218929 2560 generic.go:356] "Generic (PLEG): container finished" podID="f603ba13-e613-4b7b-bec9-c4726fbb469e" containerID="8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271" exitCode=137 Sep 16 05:09:22.219431 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.218979 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:22.219431 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.219020 2560 scope.go:117] "RemoveContainer" containerID="8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271" Sep 16 05:09:22.219431 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:22.219340 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7844-432d2d" container="test-curl-7844-432d2d" Sep 16 05:09:22.229121 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.229106 2560 scope.go:117] "RemoveContainer" containerID="8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271" Sep 16 05:09:22.229354 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:22.229337 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271\": container with ID starting with 8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271 not found: ID does not exist" containerID="8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271" Sep 16 05:09:22.229405 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.229364 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271"} err="failed to get container status \"8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271\": rpc error: code = NotFound desc = could not find container \"8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271\": container with ID starting with 8a2e31da73be8e8dde95427007d7db7b2ed6cb50377d0de944184ee77fae4271 not found: ID does not exist" Sep 16 05:09:22.230111 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:22.230091 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7844-04bfca\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" pod="opendatahub/test-curl-7844-04bfca" Sep 16 05:09:22.251763 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.251741 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx82v\" (UniqueName: \"kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v\") pod \"test-curl-7844-3923ac\" (UID: \"d15e4abb-c401-435d-8863-318be2c9c2fa\") " pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:22.261970 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.261943 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx82v\" (UniqueName: \"kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v\") pod \"test-curl-7844-3923ac\" (UID: \"d15e4abb-c401-435d-8863-318be2c9c2fa\") " pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:22.273778 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.273761 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:22.548529 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:22.548501 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7844-3923ac"] Sep 16 05:09:22.550056 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:22.550029 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd15e4abb_c401_435d_8863_318be2c9c2fa.slice/crio-afbdf585b8e79109f17ca5290a1d661032ccc3d9fd01c06e2fb84bebaf0b81eb WatchSource:0}: Error finding container afbdf585b8e79109f17ca5290a1d661032ccc3d9fd01c06e2fb84bebaf0b81eb: Status 404 returned error can't find the container with id afbdf585b8e79109f17ca5290a1d661032ccc3d9fd01c06e2fb84bebaf0b81eb Sep 16 05:09:23.122753 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.122723 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f603ba13-e613-4b7b-bec9-c4726fbb469e" path="/var/lib/kubelet/pods/f603ba13-e613-4b7b-bec9-c4726fbb469e/volumes" Sep 16 05:09:23.234577 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.234522 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-3923ac" event={"ID":"d15e4abb-c401-435d-8863-318be2c9c2fa","Type":"ContainerStarted","Data":"bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed"} Sep 16 05:09:23.234938 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.234598 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7844-3923ac" event={"ID":"d15e4abb-c401-435d-8863-318be2c9c2fa","Type":"ContainerStarted","Data":"afbdf585b8e79109f17ca5290a1d661032ccc3d9fd01c06e2fb84bebaf0b81eb"} Sep 16 05:09:23.235925 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.235899 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7844-432d2d" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" containerName="test-curl-7844-432d2d" containerID="cri-o://0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979" gracePeriod=2 Sep 16 05:09:23.252899 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.252847 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7844-3923ac" podStartSLOduration=2.252828298 podStartE2EDuration="2.252828298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:23.250644101 +0000 UTC m=+1228.764195633" watchObservedRunningTime="2026-09-16 05:09:23.252828298 +0000 UTC m=+1228.766379832" Sep 16 05:09:23.374108 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.374055 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:23.564881 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.564853 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x9vh\" (UniqueName: \"kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh\") pod \"35f49e10-92f8-4195-95c9-cdd962711b9d\" (UID: \"35f49e10-92f8-4195-95c9-cdd962711b9d\") " Sep 16 05:09:23.566823 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.566790 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh" (OuterVolumeSpecName: "kube-api-access-9x9vh") pod "35f49e10-92f8-4195-95c9-cdd962711b9d" (UID: "35f49e10-92f8-4195-95c9-cdd962711b9d"). InnerVolumeSpecName "kube-api-access-9x9vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:23.665820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.665746 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x9vh\" (UniqueName: \"kubernetes.io/projected/35f49e10-92f8-4195-95c9-cdd962711b9d-kube-api-access-9x9vh\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:23.846163 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.846130 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7844-3923ac"] Sep 16 05:09:23.853367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:23.853340 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7844-3923ac"] Sep 16 05:09:24.243948 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:24.243915 2560 generic.go:356] "Generic (PLEG): container finished" podID="35f49e10-92f8-4195-95c9-cdd962711b9d" containerID="a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b" exitCode=137 Sep 16 05:09:24.244401 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:24.243965 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:24.244401 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:24.244002 2560 scope.go:117] "RemoveContainer" containerID="a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b" Sep 16 05:09:24.244401 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:24.244234 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7844-3923ac" container="test-curl-7844-3923ac" Sep 16 05:09:24.255044 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:24.255013 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7844-d79e77\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:24.258184 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:24.258155 2560 scope.go:117] "RemoveContainer" containerID="a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b" Sep 16 05:09:24.258534 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:24.258508 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b\": container with ID starting with a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b not found: ID does not exist" containerID="a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b" Sep 16 05:09:24.258644 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:24.258565 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b"} err="failed to get container status \"a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b\": rpc error: code = NotFound desc = could not find container \"a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b\": container with ID starting with a3a073981c99b08c62f7adf9f07e5fdc4c0670a84efe15d86673c8496a0eda5b not found: ID does not exist" Sep 16 05:09:25.120034 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:25.119992 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7844-d79e77\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" pod="opendatahub/test-curl-7844-d79e77" Sep 16 05:09:25.121344 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.121323 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f49e10-92f8-4195-95c9-cdd962711b9d" path="/var/lib/kubelet/pods/35f49e10-92f8-4195-95c9-cdd962711b9d/volumes" Sep 16 05:09:25.252909 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.252876 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7844-3923ac" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" containerName="test-curl-7844-3923ac" containerID="cri-o://bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed" gracePeriod=2 Sep 16 05:09:25.409816 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.409789 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:25.480416 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.480382 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjpg5\" (UniqueName: \"kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5\") pod \"53fcd679-d16f-48ff-a7e2-4caf55cea127\" (UID: \"53fcd679-d16f-48ff-a7e2-4caf55cea127\") " Sep 16 05:09:25.482383 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.482359 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5" (OuterVolumeSpecName: "kube-api-access-xjpg5") pod "53fcd679-d16f-48ff-a7e2-4caf55cea127" (UID: "53fcd679-d16f-48ff-a7e2-4caf55cea127"). InnerVolumeSpecName "kube-api-access-xjpg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:25.581298 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:25.581266 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpg5\" (UniqueName: \"kubernetes.io/projected/53fcd679-d16f-48ff-a7e2-4caf55cea127-kube-api-access-xjpg5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:26.261400 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:26.261306 2560 generic.go:356] "Generic (PLEG): container finished" podID="53fcd679-d16f-48ff-a7e2-4caf55cea127" containerID="0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979" exitCode=137 Sep 16 05:09:26.261400 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:26.261388 2560 scope.go:117] "RemoveContainer" containerID="0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979" Sep 16 05:09:26.261400 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:26.261395 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:26.271778 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:26.271762 2560 scope.go:117] "RemoveContainer" containerID="0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979" Sep 16 05:09:26.272119 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:26.272097 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979\": container with ID starting with 0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979 not found: ID does not exist" containerID="0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979" Sep 16 05:09:26.272217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:26.272139 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979"} err="failed to get container status \"0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979\": rpc error: code = NotFound desc = could not find container \"0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979\": container with ID starting with 0c541c2be379121dff649733e9bfa092965ce63d5f64a36a85a0f5cd0ae75979 not found: ID does not exist" Sep 16 05:09:26.272873 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:26.272847 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7844-432d2d\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" pod="opendatahub/test-curl-7844-432d2d" Sep 16 05:09:27.122323 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:27.122281 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53fcd679-d16f-48ff-a7e2-4caf55cea127" path="/var/lib/kubelet/pods/53fcd679-d16f-48ff-a7e2-4caf55cea127/volumes" Sep 16 05:09:27.456870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:27.456847 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:27.499099 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:27.499067 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx82v\" (UniqueName: \"kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v\") pod \"d15e4abb-c401-435d-8863-318be2c9c2fa\" (UID: \"d15e4abb-c401-435d-8863-318be2c9c2fa\") " Sep 16 05:09:27.501151 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:27.501128 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v" (OuterVolumeSpecName: "kube-api-access-gx82v") pod "d15e4abb-c401-435d-8863-318be2c9c2fa" (UID: "d15e4abb-c401-435d-8863-318be2c9c2fa"). InnerVolumeSpecName "kube-api-access-gx82v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:27.600364 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:27.600327 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx82v\" (UniqueName: \"kubernetes.io/projected/d15e4abb-c401-435d-8863-318be2c9c2fa-kube-api-access-gx82v\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:28.279803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:28.279760 2560 generic.go:356] "Generic (PLEG): container finished" podID="d15e4abb-c401-435d-8863-318be2c9c2fa" containerID="bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed" exitCode=137 Sep 16 05:09:28.279803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:28.279799 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:28.280042 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:28.279857 2560 scope.go:117] "RemoveContainer" containerID="bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed" Sep 16 05:09:28.289487 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:28.289472 2560 scope.go:117] "RemoveContainer" containerID="bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed" Sep 16 05:09:28.289810 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:28.289790 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed\": container with ID starting with bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed not found: ID does not exist" containerID="bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed" Sep 16 05:09:28.289860 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:28.289817 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed"} err="failed to get container status \"bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed\": rpc error: code = NotFound desc = could not find container \"bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed\": container with ID starting with bb87a3063887f2696755ae4c309e40ebe6481e20e007ae12153ce0bd33291eed not found: ID does not exist" Sep 16 05:09:28.290614 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:28.290578 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7844-3923ac\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" pod="opendatahub/test-curl-7844-3923ac" Sep 16 05:09:29.122437 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:29.122399 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" path="/var/lib/kubelet/pods/d15e4abb-c401-435d-8863-318be2c9c2fa/volumes" Sep 16 05:09:35.106468 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.106435 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:09:35.107044 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.106682 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" podUID="b2637674-7a81-4cde-938e-361a28cfb006" containerName="payload-processing" containerID="cri-o://b97ba219e723835280e4b018e7e811a4b9cdfc54edb94720d04e64c8f199b3d6" gracePeriod=30 Sep 16 05:09:35.323454 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.323404 2560 generic.go:356] "Generic (PLEG): container finished" podID="b2637674-7a81-4cde-938e-361a28cfb006" containerID="b97ba219e723835280e4b018e7e811a4b9cdfc54edb94720d04e64c8f199b3d6" exitCode=0 Sep 16 05:09:35.323619 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.323475 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" event={"ID":"b2637674-7a81-4cde-938e-361a28cfb006","Type":"ContainerDied","Data":"b97ba219e723835280e4b018e7e811a4b9cdfc54edb94720d04e64c8f199b3d6"} Sep 16 05:09:35.380651 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.380531 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:09:35.485242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.485043 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v72v\" (UniqueName: \"kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v\") pod \"b2637674-7a81-4cde-938e-361a28cfb006\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " Sep 16 05:09:35.485468 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.485263 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume\") pod \"b2637674-7a81-4cde-938e-361a28cfb006\" (UID: \"b2637674-7a81-4cde-938e-361a28cfb006\") " Sep 16 05:09:35.488017 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.487981 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2637674-7a81-4cde-938e-361a28cfb006" (UID: "b2637674-7a81-4cde-938e-361a28cfb006"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:09:35.496110 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.496073 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v" (OuterVolumeSpecName: "kube-api-access-7v72v") pod "b2637674-7a81-4cde-938e-361a28cfb006" (UID: "b2637674-7a81-4cde-938e-361a28cfb006"). InnerVolumeSpecName "kube-api-access-7v72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:35.589236 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.589192 2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2637674-7a81-4cde-938e-361a28cfb006-plugins-config-volume\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:35.589236 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:35.589222 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v72v\" (UniqueName: \"kubernetes.io/projected/b2637674-7a81-4cde-938e-361a28cfb006-kube-api-access-7v72v\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:36.330824 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:36.330798 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" Sep 16 05:09:36.331202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:36.330798 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7" event={"ID":"b2637674-7a81-4cde-938e-361a28cfb006","Type":"ContainerDied","Data":"49cf25c613f1a749c0300f45bd17b16b930ee0f1ea2e888893a2caa45b5bb2c9"} Sep 16 05:09:36.331202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:36.330925 2560 scope.go:117] "RemoveContainer" containerID="b97ba219e723835280e4b018e7e811a4b9cdfc54edb94720d04e64c8f199b3d6" Sep 16 05:09:36.353939 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:36.353917 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:09:36.362095 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:36.362073 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d1f291-5f545496fc-kxvm7"] Sep 16 05:09:37.122602 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:37.122574 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2637674-7a81-4cde-938e-361a28cfb006" path="/var/lib/kubelet/pods/b2637674-7a81-4cde-938e-361a28cfb006/volumes" Sep 16 05:09:43.356084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356053 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:09:43.356488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356474 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2637674-7a81-4cde-938e-361a28cfb006" containerName="payload-processing" Sep 16 05:09:43.356488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356488 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2637674-7a81-4cde-938e-361a28cfb006" containerName="payload-processing" Sep 16 05:09:43.356580 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356497 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" containerName="test-curl-7844-3923ac" Sep 16 05:09:43.356580 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356503 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" containerName="test-curl-7844-3923ac" Sep 16 05:09:43.356668 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356611 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d15e4abb-c401-435d-8863-318be2c9c2fa" containerName="test-curl-7844-3923ac" Sep 16 05:09:43.356668 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.356623 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2637674-7a81-4cde-938e-361a28cfb006" containerName="payload-processing" Sep 16 05:09:43.361123 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.361108 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.363907 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.363885 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3784e2-openshift-default-dockercfg-4xjnp\"" Sep 16 05:09:43.381267 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.381242 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:09:43.478418 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478379 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478643 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478422 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478643 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478643 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478483 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478643 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478505 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478643 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478614 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.478884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.478764 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdd44\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580187 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580217 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580243 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580332 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580418 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580764 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580458 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdd44\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580764 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580715 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580885 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580816 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.580885 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580834 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.581002 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.580925 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.581422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.581401 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.582571 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.582534 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.583024 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.583004 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.590883 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.590856 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.590883 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.590872 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdd44\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44\") pod \"e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.671698 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.671615 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:43.765084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.765054 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:09:43.978302 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:09:43.978273 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a00bc64_dbf8_497a_8f86_2650ceff85de.slice/crio-f98a1e362bf4b7cf5fcea8d76a5ff893e0782367da67cc09fb1e1481b881aee1 WatchSource:0}: Error finding container f98a1e362bf4b7cf5fcea8d76a5ff893e0782367da67cc09fb1e1481b881aee1: Status 404 returned error can't find the container with id f98a1e362bf4b7cf5fcea8d76a5ff893e0782367da67cc09fb1e1481b881aee1 Sep 16 05:09:43.980763 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.980730 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:09:43.980856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.980794 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:09:43.980856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.980823 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:09:43.981491 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:43.981463 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:09:44.382539 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:44.382501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerStarted","Data":"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490"} Sep 16 05:09:44.382899 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:44.382572 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerStarted","Data":"f98a1e362bf4b7cf5fcea8d76a5ff893e0782367da67cc09fb1e1481b881aee1"} Sep 16 05:09:44.470164 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:44.470115 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podStartSLOduration=1.470097234 podStartE2EDuration="1.470097234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:09:44.467945905 +0000 UTC m=+1249.981497470" watchObservedRunningTime="2026-09-16 05:09:44.470097234 +0000 UTC m=+1249.983648767" Sep 16 05:09:44.672543 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:44.672462 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:45.678057 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:45.678028 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:46.395252 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:46.395218 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:46.396171 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:46.396146 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:09:58.784241 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:58.784202 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" podUID="6cccd0c2-3749-46ef-893b-503b1097c7b2" containerName="main" containerID="cri-o://ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de" gracePeriod=45 Sep 16 05:09:59.042142 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.042076 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:59.152235 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152205 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b7r7\" (UniqueName: \"kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152248 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152282 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152340 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152356 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir\") pod \"6cccd0c2-3749-46ef-893b-503b1097c7b2\" (UID: \"6cccd0c2-3749-46ef-893b-503b1097c7b2\") " Sep 16 05:09:59.152793 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152615 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:09:59.152793 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152630 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home" (OuterVolumeSpecName: "home") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:09:59.152908 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152824 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.152908 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152845 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.153014 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.152915 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:09:59.154734 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.154704 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:09:59.154967 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.154938 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm" (OuterVolumeSpecName: "dshm") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:09:59.155068 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.154975 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7" (OuterVolumeSpecName: "kube-api-access-5b7r7") pod "6cccd0c2-3749-46ef-893b-503b1097c7b2" (UID: "6cccd0c2-3749-46ef-893b-503b1097c7b2"). InnerVolumeSpecName "kube-api-access-5b7r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:09:59.253786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.253751 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b7r7\" (UniqueName: \"kubernetes.io/projected/6cccd0c2-3749-46ef-893b-503b1097c7b2-kube-api-access-5b7r7\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.253786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.253778 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.253786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.253789 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cccd0c2-3749-46ef-893b-503b1097c7b2-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.253786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.253797 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cccd0c2-3749-46ef-893b-503b1097c7b2-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:09:59.479005 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.478967 2560 generic.go:356] "Generic (PLEG): container finished" podID="6cccd0c2-3749-46ef-893b-503b1097c7b2" containerID="ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de" exitCode=0 Sep 16 05:09:59.479005 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.479009 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" Sep 16 05:09:59.479251 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.479028 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" event={"ID":"6cccd0c2-3749-46ef-893b-503b1097c7b2","Type":"ContainerDied","Data":"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de"} Sep 16 05:09:59.479251 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.479065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h" event={"ID":"6cccd0c2-3749-46ef-893b-503b1097c7b2","Type":"ContainerDied","Data":"2771b1d68ca2091cce99023bdb8984608d5126489c9a54a049c7f18aeb46d45f"} Sep 16 05:09:59.479251 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.479088 2560 scope.go:117] "RemoveContainer" containerID="ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de" Sep 16 05:09:59.488788 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.488766 2560 scope.go:117] "RemoveContainer" containerID="ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de" Sep 16 05:09:59.489063 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:09:59.489043 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de\": container with ID starting with ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de not found: ID does not exist" containerID="ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de" Sep 16 05:09:59.489109 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.489072 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de"} err="failed to get container status \"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de\": rpc error: code = NotFound desc = could not find container \"ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de\": container with ID starting with ac1bf36999ab8ee97e504c175b4b14830db559b14c37c0bdaf3212b86a0977de not found: ID does not exist" Sep 16 05:09:59.501616 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.501587 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:09:59.507669 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:09:59.507647 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-748b01/noten-model-748b01-kserve-7769bd86ff-29c2h"] Sep 16 05:10:01.129173 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:01.125127 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cccd0c2-3749-46ef-893b-503b1097c7b2" path="/var/lib/kubelet/pods/6cccd0c2-3749-46ef-893b-503b1097c7b2/volumes" Sep 16 05:10:17.263227 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.263195 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:10:17.263696 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.263669 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cccd0c2-3749-46ef-893b-503b1097c7b2" containerName="main" Sep 16 05:10:17.263696 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.263683 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cccd0c2-3749-46ef-893b-503b1097c7b2" containerName="main" Sep 16 05:10:17.263771 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.263764 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cccd0c2-3749-46ef-893b-503b1097c7b2" containerName="main" Sep 16 05:10:17.268206 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.268191 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.271927 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.271909 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-10ee79-openshift-default-dockercfg-njbs6\"" Sep 16 05:10:17.292570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.292515 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:10:17.343390 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343348 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343390 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343421 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343542 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343642 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343636 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343808 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343668 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrmf\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.343808 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.343705 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.436705 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.436665 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 16 05:10:17.436896 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.436720 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 05:10:17.444311 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444400 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444443 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444443 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444426 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444561 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444561 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444483 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444561 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444515 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444711 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444711 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrmf\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.444826 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.444727 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.445214 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.445190 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.445318 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.445240 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.445318 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.445262 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.445318 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.445274 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.446708 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.446690 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.446962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.446942 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.453345 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.453322 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.453429 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.453381 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrmf\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf\") pod \"e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.578416 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.578386 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:17.861569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.861519 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:10:17.862106 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:10:17.862071 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3904c14b_5e48_4338_8f3f_ba52d63f5312.slice/crio-dccb9e52d421c0931b92bd74f9deec221b90cd8112877817298419f1bb7a8600 WatchSource:0}: Error finding container dccb9e52d421c0931b92bd74f9deec221b90cd8112877817298419f1bb7a8600: Status 404 returned error can't find the container with id dccb9e52d421c0931b92bd74f9deec221b90cd8112877817298419f1bb7a8600 Sep 16 05:10:17.864961 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.864919 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:17.865041 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.865008 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:17.865087 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:17.865054 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:18.618200 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.618170 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerID="96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490" exitCode=0 Sep 16 05:10:18.618670 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.618240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerDied","Data":"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490"} Sep 16 05:10:18.618728 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.618712 2560 scope.go:117] "RemoveContainer" containerID="96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490" Sep 16 05:10:18.620083 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.620049 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:18.620196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.620119 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerStarted","Data":"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac"} Sep 16 05:10:18.620196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.620137 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:18.620196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.620156 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerStarted","Data":"dccb9e52d421c0931b92bd74f9deec221b90cd8112877817298419f1bb7a8600"} Sep 16 05:10:18.620196 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.620190 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:18.663870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:18.663406 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podStartSLOduration=1.66338397 podStartE2EDuration="1.66338397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:10:18.659991837 +0000 UTC m=+1284.173543373" watchObservedRunningTime="2026-09-16 05:10:18.66338397 +0000 UTC m=+1284.176935505" Sep 16 05:10:19.579611 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.579569 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:19.585773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.585741 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:19.628713 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.628674 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerStarted","Data":"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269"} Sep 16 05:10:19.629099 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.628869 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:19.629748 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.629730 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:10:19.672272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.672227 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:10:19.673359 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.673240 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:19.673565 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:19.673394 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:20.672805 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:20.672761 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:20.673306 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:20.672831 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:21.672928 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:21.672883 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:21.673430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:21.672975 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:22.672678 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:22.672642 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:22.673348 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:22.672964 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:23.673086 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:23.673047 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:23.673501 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:23.673465 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:24.672606 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:24.672564 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 16 05:10:24.672790 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:24.672639 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 16 05:10:25.676253 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:25.676225 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:10:25.676701 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:25.676517 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:10:25.677174 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:25.677155 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:10:26.429152 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.429119 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:10:26.432741 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.432722 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.435577 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.435535 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-10ee79-serving-cert\"" Sep 16 05:10:26.447103 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.447078 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:10:26.553367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.553330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ggr\" (UniqueName: \"kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.553367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.553370 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.553609 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.553443 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.553609 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.553466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.654859 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.654820 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ggr\" (UniqueName: \"kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.654859 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.654865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.655124 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.654905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.655124 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.654927 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.655124 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:10:26.655066 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-serving-cert: secret "maas-api-e2e-shared-b-w3-10ee79-serving-cert" not found Sep 16 05:10:26.655283 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:10:26.655144 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls podName:42315721-8b38-4fb1-8bb9-11b51e73748a nodeName:}" failed. No retries permitted until 2026-09-16 05:10:27.155122779 +0000 UTC m=+1292.668674307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" (UID: "42315721-8b38-4fb1-8bb9-11b51e73748a") : secret "maas-api-e2e-shared-b-w3-10ee79-serving-cert" not found Sep 16 05:10:26.655695 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.655669 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.657540 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.657514 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:26.666311 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:26.666290 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ggr\" (UniqueName: \"kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:27.161279 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:27.161220 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:27.166532 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:27.166422 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:27.344714 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:27.344672 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:27.621383 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:27.621348 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:10:27.623242 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:10:27.623215 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42315721_8b38_4fb1_8bb9_11b51e73748a.slice/crio-f72b87cb8b6dbcea1d50d381931f6e0b1b9c57c8cf44fd4a5c8bb8ab5999ed34 WatchSource:0}: Error finding container f72b87cb8b6dbcea1d50d381931f6e0b1b9c57c8cf44fd4a5c8bb8ab5999ed34: Status 404 returned error can't find the container with id f72b87cb8b6dbcea1d50d381931f6e0b1b9c57c8cf44fd4a5c8bb8ab5999ed34 Sep 16 05:10:27.686447 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:27.686410 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" event={"ID":"42315721-8b38-4fb1-8bb9-11b51e73748a","Type":"ContainerStarted","Data":"f72b87cb8b6dbcea1d50d381931f6e0b1b9c57c8cf44fd4a5c8bb8ab5999ed34"} Sep 16 05:10:28.695536 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:28.695500 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" event={"ID":"42315721-8b38-4fb1-8bb9-11b51e73748a","Type":"ContainerStarted","Data":"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0"} Sep 16 05:10:28.695957 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:28.695702 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:28.719188 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:28.719133 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" podStartSLOduration=2.374414941 podStartE2EDuration="2.719114863s" totalImagesPullingTime="344.699922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:10:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:10:28.716801483 +0000 UTC m=+1294.230353018" watchObservedRunningTime="2026-09-16 05:10:28.719114863 +0000 UTC m=+1294.232666427" Sep 16 05:10:34.707569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:34.707515 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:10:48.458326 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.458289 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:10:48.464601 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.464576 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.467292 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.467267 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-766a69c0\"/\"openshift-service-ca.crt\"" Sep 16 05:10:48.468689 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.468485 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-766a69c0\"/\"kube-root-ca.crt\"" Sep 16 05:10:48.468818 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.468732 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-766a69c0\"/\"default-dockercfg-66lbq\"" Sep 16 05:10:48.468818 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.468748 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-766a69c0\"/\"e2e-lifecycle-model-766a69c0-kserve-self-signed-certs\"" Sep 16 05:10:48.470821 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.470803 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:10:48.571747 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571711 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.571912 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.571912 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4gzn\" (UniqueName: \"kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.571912 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571860 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.571912 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571885 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.572055 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.571922 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.672710 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672673 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.672901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672746 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4gzn\" (UniqueName: \"kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.672901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.672901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672820 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.672901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672872 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.673120 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.672908 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.673167 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.673116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.673202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.673175 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.673202 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.673191 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.675166 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.675135 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.675295 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.675278 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.682445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.682418 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4gzn\" (UniqueName: \"kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn\") pod \"e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:48.775136 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:48.775060 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:49.057785 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:49.057754 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:10:49.059051 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:10:49.059018 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f0bd9e9_ce84_4989_9ffc_77086b2f4a50.slice/crio-80e429f7e18a0c4f65059d4a19258070d4fd06d25c3566a30d95ec90a3ee10f5 WatchSource:0}: Error finding container 80e429f7e18a0c4f65059d4a19258070d4fd06d25c3566a30d95ec90a3ee10f5: Status 404 returned error can't find the container with id 80e429f7e18a0c4f65059d4a19258070d4fd06d25c3566a30d95ec90a3ee10f5 Sep 16 05:10:49.839446 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:49.839408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" event={"ID":"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50","Type":"ContainerStarted","Data":"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6"} Sep 16 05:10:49.839876 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:49.839461 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" event={"ID":"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50","Type":"ContainerStarted","Data":"80e429f7e18a0c4f65059d4a19258070d4fd06d25c3566a30d95ec90a3ee10f5"} Sep 16 05:10:49.859092 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:49.859036 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" podStartSLOduration=1.859018017 podStartE2EDuration="1.859018017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:10:49.857583828 +0000 UTC m=+1315.371135366" watchObservedRunningTime="2026-09-16 05:10:49.859018017 +0000 UTC m=+1315.372569550" Sep 16 05:10:58.775305 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:58.775270 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:58.775305 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:58.775316 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:58.788004 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:58.787981 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:58.929999 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:58.929966 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:10:59.919292 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.919257 2560 generic.go:356] "Generic (PLEG): container finished" podID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerID="f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac" exitCode=0 Sep 16 05:10:59.919765 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.919334 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerDied","Data":"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac"} Sep 16 05:10:59.919897 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.919880 2560 scope.go:117] "RemoveContainer" containerID="f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac" Sep 16 05:10:59.921021 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.920991 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:59.921114 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.921060 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:10:59.921114 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:10:59.921091 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:11:00.927441 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:00.927409 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerStarted","Data":"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4"} Sep 16 05:11:01.578714 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:01.578677 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:11:01.580264 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:01.580227 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:01.580402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:01.580294 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:02.579038 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:02.578998 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:02.579421 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:02.579081 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:03.579113 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:03.579071 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:03.579507 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:03.579135 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:04.579288 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:04.579244 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:04.579714 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:04.579337 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:04.629500 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:04.629467 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:11:05.579128 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:05.579078 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:05.579341 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:05.579151 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:06.578868 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:06.578829 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 16 05:11:06.579090 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:06.578896 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 16 05:11:07.583525 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:07.583494 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:11:07.584766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:07.584745 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:11:38.232959 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:38.232927 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:11:53.252349 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.252282 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" podUID="2fd23d95-8741-453d-bd73-b96d525623ba" containerName="main" containerID="cri-o://5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf" gracePeriod=45 Sep 16 05:11:53.518021 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.517965 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:11:53.556161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556137 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556304 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556206 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556357 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556330 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556417 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556478 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556438 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdzvw\" (UniqueName: \"kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556537 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556469 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home" (OuterVolumeSpecName: "home") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:11:53.556537 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556481 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir\") pod \"2fd23d95-8741-453d-bd73-b96d525623ba\" (UID: \"2fd23d95-8741-453d-bd73-b96d525623ba\") " Sep 16 05:11:53.556852 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556790 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:11:53.556970 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556929 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:11:53.557039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.556983 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:53.557039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.557005 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:53.558753 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.558726 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:11:53.558840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.558753 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw" (OuterVolumeSpecName: "kube-api-access-rdzvw") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "kube-api-access-rdzvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:11:53.558840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.558780 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm" (OuterVolumeSpecName: "dshm") pod "2fd23d95-8741-453d-bd73-b96d525623ba" (UID: "2fd23d95-8741-453d-bd73-b96d525623ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:11:53.657826 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.657796 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fd23d95-8741-453d-bd73-b96d525623ba-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:53.657826 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.657821 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:53.657826 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.657831 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzvw\" (UniqueName: \"kubernetes.io/projected/2fd23d95-8741-453d-bd73-b96d525623ba-kube-api-access-rdzvw\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:53.658039 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:53.657840 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fd23d95-8741-453d-bd73-b96d525623ba-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:11:54.292611 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.292572 2560 generic.go:356] "Generic (PLEG): container finished" podID="2fd23d95-8741-453d-bd73-b96d525623ba" containerID="5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf" exitCode=0 Sep 16 05:11:54.293049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.292629 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" Sep 16 05:11:54.293049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.292646 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" event={"ID":"2fd23d95-8741-453d-bd73-b96d525623ba","Type":"ContainerDied","Data":"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf"} Sep 16 05:11:54.293049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.292684 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz" event={"ID":"2fd23d95-8741-453d-bd73-b96d525623ba","Type":"ContainerDied","Data":"715b09c3ca266c11d25884ab1edb8d6a058c46ff661c7e56271d36e944774eba"} Sep 16 05:11:54.293049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.292707 2560 scope.go:117] "RemoveContainer" containerID="5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf" Sep 16 05:11:54.302649 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.302629 2560 scope.go:117] "RemoveContainer" containerID="5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf" Sep 16 05:11:54.302926 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:11:54.302905 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf\": container with ID starting with 5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf not found: ID does not exist" containerID="5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf" Sep 16 05:11:54.303019 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.302931 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf"} err="failed to get container status \"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf\": rpc error: code = NotFound desc = could not find container \"5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf\": container with ID starting with 5d6f165230eb9e32d2363cacd1a1dc766e5ae8529736f0c969e7b028a95569bf not found: ID does not exist" Sep 16 05:11:54.313138 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.313112 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:11:54.316226 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:54.316206 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/e2e-conflict-a-2a88c17b-kserve-68dc648cfc-cfszz"] Sep 16 05:11:55.127426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:11:55.127385 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fd23d95-8741-453d-bd73-b96d525623ba" path="/var/lib/kubelet/pods/2fd23d95-8741-453d-bd73-b96d525623ba/volumes" Sep 16 05:12:18.746559 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:18.746505 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:12:33.766315 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:33.766269 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" podUID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" containerName="main" containerID="cri-o://71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6" gracePeriod=45 Sep 16 05:12:34.031031 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.030970 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:12:34.158835 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158802 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4gzn\" (UniqueName: \"kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159034 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158861 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159034 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158877 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159034 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158902 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159034 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158933 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159034 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.158975 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home\") pod \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\" (UID: \"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50\") " Sep 16 05:12:34.159317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.159175 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:12:34.159317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.159206 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:12:34.159317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.159290 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home" (OuterVolumeSpecName: "home") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:12:34.159480 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.159353 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.159480 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.159367 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.161128 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.161104 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn" (OuterVolumeSpecName: "kube-api-access-x4gzn") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "kube-api-access-x4gzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:12:34.161452 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.161435 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm" (OuterVolumeSpecName: "dshm") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:12:34.161513 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.161452 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" (UID: "9f0bd9e9-ce84-4989-9ffc-77086b2f4a50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:12:34.260786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.260739 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4gzn\" (UniqueName: \"kubernetes.io/projected/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-kube-api-access-x4gzn\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.260786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.260780 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.260786 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.260795 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.260990 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.260808 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:12:34.544940 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.544900 2560 generic.go:356] "Generic (PLEG): container finished" podID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" containerID="71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6" exitCode=0 Sep 16 05:12:34.545140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.544951 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" Sep 16 05:12:34.545140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.544991 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" event={"ID":"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50","Type":"ContainerDied","Data":"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6"} Sep 16 05:12:34.545140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.545048 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv" event={"ID":"9f0bd9e9-ce84-4989-9ffc-77086b2f4a50","Type":"ContainerDied","Data":"80e429f7e18a0c4f65059d4a19258070d4fd06d25c3566a30d95ec90a3ee10f5"} Sep 16 05:12:34.545140 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.545072 2560 scope.go:117] "RemoveContainer" containerID="71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6" Sep 16 05:12:34.555076 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.555058 2560 scope.go:117] "RemoveContainer" containerID="71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6" Sep 16 05:12:34.555335 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:12:34.555318 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6\": container with ID starting with 71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6 not found: ID does not exist" containerID="71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6" Sep 16 05:12:34.555399 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.555342 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6"} err="failed to get container status \"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6\": rpc error: code = NotFound desc = could not find container \"71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6\": container with ID starting with 71996966429a2848f9a788434d5466efef3149c1a2addfb9f5c8ae07c6e6d0f6 not found: ID does not exist" Sep 16 05:12:34.567749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.567729 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:12:34.571384 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:34.571356 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-766a69c0/e2e-lifecycle-model-766a69c0-kserve-9d94dbd8c-47jkv"] Sep 16 05:12:35.121393 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:12:35.121359 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" path="/var/lib/kubelet/pods/9f0bd9e9-ce84-4989-9ffc-77086b2f4a50/volumes" Sep 16 05:13:37.148832 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.148798 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:13:37.149296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149281 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" containerName="main" Sep 16 05:13:37.149343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149298 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" containerName="main" Sep 16 05:13:37.149343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149318 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fd23d95-8741-453d-bd73-b96d525623ba" containerName="main" Sep 16 05:13:37.149343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149324 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fd23d95-8741-453d-bd73-b96d525623ba" containerName="main" Sep 16 05:13:37.149436 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149403 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fd23d95-8741-453d-bd73-b96d525623ba" containerName="main" Sep 16 05:13:37.149436 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.149411 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0bd9e9-ce84-4989-9ffc-77086b2f4a50" containerName="main" Sep 16 05:13:37.152726 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.152710 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.156887 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.156775 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"openshift-service-ca.crt\"" Sep 16 05:13:37.157077 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.156951 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"embedding-w0-ee7a60-kserve-self-signed-certs\"" Sep 16 05:13:37.157077 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.156955 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"default-dockercfg-jzbds\"" Sep 16 05:13:37.157282 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.157253 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ee7a60\"/\"kube-root-ca.crt\"" Sep 16 05:13:37.161132 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.161109 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:13:37.197363 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197336 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.197476 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197376 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.197476 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197403 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.197591 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.197591 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197513 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbf4t\" (UniqueName: \"kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.197691 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.197591 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298664 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298630 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298710 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298763 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298786 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.298841 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.298803 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbf4t\" (UniqueName: \"kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.299127 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.299073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.299127 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.299114 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.299233 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.299160 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.301105 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.301079 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.301374 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.301355 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.306325 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.306297 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbf4t\" (UniqueName: \"kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t\") pod \"embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.462797 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.462709 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:37.751123 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.751088 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:13:37.752353 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:13:37.752327 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c1df1a8_3eff_47c5_909a_2d47af0e8a66.slice/crio-2ce1a5a0d17282b6f0d4994c7553af6081434e75cc6c2052dcc19314aebd86d3 WatchSource:0}: Error finding container 2ce1a5a0d17282b6f0d4994c7553af6081434e75cc6c2052dcc19314aebd86d3: Status 404 returned error can't find the container with id 2ce1a5a0d17282b6f0d4994c7553af6081434e75cc6c2052dcc19314aebd86d3 Sep 16 05:13:37.940667 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.940622 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" event={"ID":"7c1df1a8-3eff-47c5-909a-2d47af0e8a66","Type":"ContainerStarted","Data":"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432"} Sep 16 05:13:37.940876 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.940678 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" event={"ID":"7c1df1a8-3eff-47c5-909a-2d47af0e8a66","Type":"ContainerStarted","Data":"2ce1a5a0d17282b6f0d4994c7553af6081434e75cc6c2052dcc19314aebd86d3"} Sep 16 05:13:37.961045 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:37.960990 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" podStartSLOduration=0.960973539 podStartE2EDuration="960.973539ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:13:37.960133579 +0000 UTC m=+1483.473685149" watchObservedRunningTime="2026-09-16 05:13:37.960973539 +0000 UTC m=+1483.474525072" Sep 16 05:13:47.463621 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:47.463581 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:47.464181 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:47.463645 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:47.476236 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:47.476209 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:48.017315 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:48.017284 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:13:55.128520 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:55.128487 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:13:55.128975 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:55.128589 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:13:55.131324 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:55.131301 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:13:55.131324 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:13:55.131313 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:14:39.747030 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:39.746994 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:14:52.949570 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:52.949500 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:14:52.950108 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:52.949752 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" podUID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" containerName="payload-pre-processing" containerID="cri-o://32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72" gracePeriod=30 Sep 16 05:14:53.224447 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.224423 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:14:53.293887 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.293858 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgdt9\" (UniqueName: \"kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9\") pod \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " Sep 16 05:14:53.294087 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.293901 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume\") pod \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\" (UID: \"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6\") " Sep 16 05:14:53.294382 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.294352 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" (UID: "fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:14:53.296177 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.296149 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9" (OuterVolumeSpecName: "kube-api-access-cgdt9") pod "fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" (UID: "fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6"). InnerVolumeSpecName "kube-api-access-cgdt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:14:53.395625 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.395595 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdt9\" (UniqueName: \"kubernetes.io/projected/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-kube-api-access-cgdt9\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:53.395625 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.395623 2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6-plugins-config-volume\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:53.425663 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.425630 2560 generic.go:356] "Generic (PLEG): container finished" podID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" containerID="32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72" exitCode=0 Sep 16 05:14:53.425815 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.425743 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" Sep 16 05:14:53.425815 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.425781 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" event={"ID":"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6","Type":"ContainerDied","Data":"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72"} Sep 16 05:14:53.425890 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.425818 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq" event={"ID":"fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6","Type":"ContainerDied","Data":"b43ea2f4e542b4843a24ea3f303e4496b3cf14951837e50b46ef1d8b82ce07cd"} Sep 16 05:14:53.425890 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.425840 2560 scope.go:117] "RemoveContainer" containerID="32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72" Sep 16 05:14:53.435133 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.435114 2560 scope.go:117] "RemoveContainer" containerID="32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72" Sep 16 05:14:53.435414 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:14:53.435396 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72\": container with ID starting with 32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72 not found: ID does not exist" containerID="32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72" Sep 16 05:14:53.435467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.435421 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72"} err="failed to get container status \"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72\": rpc error: code = NotFound desc = could not find container \"32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72\": container with ID starting with 32eb01bbe9e86b59a5c129fb24b83487ea28ce300766875d339ba497107b5a72 not found: ID does not exist" Sep 16 05:14:53.450243 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.450217 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:14:53.455868 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:53.455844 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ee7a60-686f96c9c4-4phqq"] Sep 16 05:14:54.767754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:54.767709 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" podUID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" containerName="main" containerID="cri-o://6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432" gracePeriod=45 Sep 16 05:14:55.031718 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.031637 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:14:55.112395 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112367 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112395 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112397 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112608 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112434 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112608 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112517 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbf4t\" (UniqueName: \"kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112608 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112567 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112615 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm\") pod \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\" (UID: \"7c1df1a8-3eff-47c5-909a-2d47af0e8a66\") " Sep 16 05:14:55.112749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112684 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:14:55.112749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112718 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:14:55.112749 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.112734 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home" (OuterVolumeSpecName: "home") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:14:55.113485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.113070 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.113485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.113092 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.113485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.113104 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.114829 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.114803 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:14:55.115058 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.115040 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm" (OuterVolumeSpecName: "dshm") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:14:55.115148 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.115132 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t" (OuterVolumeSpecName: "kube-api-access-gbf4t") pod "7c1df1a8-3eff-47c5-909a-2d47af0e8a66" (UID: "7c1df1a8-3eff-47c5-909a-2d47af0e8a66"). InnerVolumeSpecName "kube-api-access-gbf4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:14:55.128407 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.128377 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" path="/var/lib/kubelet/pods/fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6/volumes" Sep 16 05:14:55.214322 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.214293 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbf4t\" (UniqueName: \"kubernetes.io/projected/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-kube-api-access-gbf4t\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.214322 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.214323 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.214507 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.214334 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1df1a8-3eff-47c5-909a-2d47af0e8a66-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:14:55.440925 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.440895 2560 generic.go:356] "Generic (PLEG): container finished" podID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" containerID="6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432" exitCode=0 Sep 16 05:14:55.441084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.440948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" event={"ID":"7c1df1a8-3eff-47c5-909a-2d47af0e8a66","Type":"ContainerDied","Data":"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432"} Sep 16 05:14:55.441084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.440961 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" Sep 16 05:14:55.441084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.440978 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r" event={"ID":"7c1df1a8-3eff-47c5-909a-2d47af0e8a66","Type":"ContainerDied","Data":"2ce1a5a0d17282b6f0d4994c7553af6081434e75cc6c2052dcc19314aebd86d3"} Sep 16 05:14:55.441084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.440996 2560 scope.go:117] "RemoveContainer" containerID="6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432" Sep 16 05:14:55.451103 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.451085 2560 scope.go:117] "RemoveContainer" containerID="6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432" Sep 16 05:14:55.451366 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:14:55.451346 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432\": container with ID starting with 6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432 not found: ID does not exist" containerID="6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432" Sep 16 05:14:55.451457 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.451374 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432"} err="failed to get container status \"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432\": rpc error: code = NotFound desc = could not find container \"6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432\": container with ID starting with 6d7d55788fb4fc4bb3bddbbf460216ebd3e0424d9acfa9b4ea8a37af822dd432 not found: ID does not exist" Sep 16 05:14:55.462462 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.462434 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:14:55.465794 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:55.465773 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ee7a60/embedding-w0-ee7a60-kserve-7c47fc86b8-qhk9r"] Sep 16 05:14:57.121640 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:14:57.121612 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" path="/var/lib/kubelet/pods/7c1df1a8-3eff-47c5-909a-2d47af0e8a66/volumes" Sep 16 05:15:00.008565 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.008522 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:15:00.008954 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.008745 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" containerID="cri-o://510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8" gracePeriod=30 Sep 16 05:15:00.145655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.145623 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb"] Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147140 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" containerName="payload-pre-processing" Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147173 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" containerName="payload-pre-processing" Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147231 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" containerName="main" Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147240 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" containerName="main" Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147479 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa50c0ce-6901-4ae5-bfc3-7e94d0b86cc6" containerName="payload-pre-processing" Sep 16 05:15:00.149572 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.147504 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c1df1a8-3eff-47c5-909a-2d47af0e8a66" containerName="main" Sep 16 05:15:00.152673 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.152012 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" Sep 16 05:15:00.157205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.157176 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-sm548\"" Sep 16 05:15:00.167670 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.167648 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb"] Sep 16 05:15:00.168858 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.168834 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" Sep 16 05:15:00.172656 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.172634 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx"] Sep 16 05:15:00.178807 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.178503 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" Sep 16 05:15:00.181212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.181186 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx"] Sep 16 05:15:00.212925 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.212901 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" Sep 16 05:15:00.465963 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.465939 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb"] Sep 16 05:15:00.467485 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:15:00.467455 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bb853f_deec_451f_a44f_8257f08f9353.slice/crio-ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398 WatchSource:0}: Error finding container ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398: Status 404 returned error can't find the container with id ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398 Sep 16 05:15:00.469631 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.469463 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 05:15:00.475930 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.475880 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" event={"ID":"a1bb853f-deec-451f-a44f-8257f08f9353","Type":"ContainerStarted","Data":"ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398"} Sep 16 05:15:00.520319 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.520292 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx"] Sep 16 05:15:00.521975 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:15:00.521947 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7a329f_e9ef_4b93_9bca_5bb2aaa778fe.slice/crio-b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e WatchSource:0}: Error finding container b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e: Status 404 returned error can't find the container with id b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e Sep 16 05:15:00.774592 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.774541 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:15:00.779655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.779633 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.782114 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.782093 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-113278-openshift-default-dockercfg-7fbmj\"" Sep 16 05:15:00.790178 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.790153 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:15:00.876105 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876291 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876114 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876291 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876203 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876291 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876242 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876291 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876315 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876361 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876429 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjj4\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.876488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.876466 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977037 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjj4\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977135 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977165 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977220 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977291 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977324 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977372 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977666 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977640 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977693 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.977836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.977782 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.978097 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.978073 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.979794 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.979767 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.980015 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.979997 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.985233 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.985213 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:00.985439 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:00.985422 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjj4\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4\") pod \"e2e-api-b-113278-openshift-default-645647b4cd-h56mm\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:01.095452 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.095415 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:01.376314 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.376281 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:15:01.377779 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:15:01.377749 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdd7ae2_70e9_4434_8df1_7d04f67559f8.slice/crio-574a7e9cdcc297ab093cba0d9d930ce000b9cc480c3109c683de8f547ccb6fbe WatchSource:0}: Error finding container 574a7e9cdcc297ab093cba0d9d930ce000b9cc480c3109c683de8f547ccb6fbe: Status 404 returned error can't find the container with id 574a7e9cdcc297ab093cba0d9d930ce000b9cc480c3109c683de8f547ccb6fbe Sep 16 05:15:01.380676 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.380633 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:01.380795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.380722 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:01.380795 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.380756 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:01.486780 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.486651 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerStarted","Data":"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb"} Sep 16 05:15:01.486780 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.486708 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerStarted","Data":"574a7e9cdcc297ab093cba0d9d930ce000b9cc480c3109c683de8f547ccb6fbe"} Sep 16 05:15:01.488595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.488524 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" event={"ID":"bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe","Type":"ContainerStarted","Data":"b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e"} Sep 16 05:15:01.517865 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:01.517810 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podStartSLOduration=1.51779324 podStartE2EDuration="1.51779324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:01.514634401 +0000 UTC m=+1567.028185963" watchObservedRunningTime="2026-09-16 05:15:01.51779324 +0000 UTC m=+1567.031344774" Sep 16 05:15:02.096695 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:02.096607 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:02.098950 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:02.098648 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:02.098950 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:02.098714 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:03.100332 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.100254 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:03.507374 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.507273 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" event={"ID":"bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe","Type":"ContainerStarted","Data":"67cb618db79a724ae435e77ea959c324314ed8bbf032bf285d58b74dccc7b7a1"} Sep 16 05:15:03.512255 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.512220 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" event={"ID":"a1bb853f-deec-451f-a44f-8257f08f9353","Type":"ContainerStarted","Data":"6802e3489f67752b36abe514616101b4741291c33c099a1ddf6a4d92d2c8aaa0"} Sep 16 05:15:03.513445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.513409 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:03.520134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.520102 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:03.534254 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.534202 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" podStartSLOduration=1.201571736 podStartE2EDuration="3.534185316s" totalImagesPullingTime="2.33261358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:03.530862747 +0000 UTC m=+1569.044414280" watchObservedRunningTime="2026-09-16 05:15:03.534185316 +0000 UTC m=+1569.047736849" Sep 16 05:15:03.591523 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:03.591469 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" podStartSLOduration=1.200589224 podStartE2EDuration="3.591449588s" totalImagesPullingTime="2.390860364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:15:03.587642223 +0000 UTC m=+1569.101193770" watchObservedRunningTime="2026-09-16 05:15:03.591449588 +0000 UTC m=+1569.105001124" Sep 16 05:15:04.520402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:04.520358 2560 generic.go:356] "Generic (PLEG): container finished" podID="a1bb853f-deec-451f-a44f-8257f08f9353" containerID="6802e3489f67752b36abe514616101b4741291c33c099a1ddf6a4d92d2c8aaa0" exitCode=0 Sep 16 05:15:04.520958 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:04.520447 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" event={"ID":"a1bb853f-deec-451f-a44f-8257f08f9353","Type":"ContainerDied","Data":"6802e3489f67752b36abe514616101b4741291c33c099a1ddf6a4d92d2c8aaa0"} Sep 16 05:15:04.522110 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:04.522086 2560 generic.go:356] "Generic (PLEG): container finished" podID="bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe" containerID="67cb618db79a724ae435e77ea959c324314ed8bbf032bf285d58b74dccc7b7a1" exitCode=0 Sep 16 05:15:04.522217 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:04.522169 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" event={"ID":"bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe","Type":"ContainerDied","Data":"67cb618db79a724ae435e77ea959c324314ed8bbf032bf285d58b74dccc7b7a1"} Sep 16 05:15:05.264397 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.264367 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:15:05.325866 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.325828 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.325884 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.325926 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326157 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326047 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326157 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326096 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326157 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326134 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86q5x\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326318 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326214 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326306 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:15:05.326378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326323 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.326378 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326326 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:15:05.326574 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326440 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:15:05.326574 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326504 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:15:05.326856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326811 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.326856 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326839 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.326983 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326854 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.326983 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.326868 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.328376 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.328348 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x" (OuterVolumeSpecName: "kube-api-access-86q5x") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "kube-api-access-86q5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:15:05.328488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.328371 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:15:05.328488 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.328467 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:15:05.328680 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.328662 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:15:05.428118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428085 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs\") pod \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\" (UID: \"ef6dd537-7a92-4d81-8277-de36eafb5dfb\") " Sep 16 05:15:05.428333 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428312 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef6dd537-7a92-4d81-8277-de36eafb5dfb" (UID: "ef6dd537-7a92-4d81-8277-de36eafb5dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:15:05.428467 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428453 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.428505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428470 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.428505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428479 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.428505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428487 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6dd537-7a92-4d81-8277-de36eafb5dfb-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.428505 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.428495 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86q5x\" (UniqueName: \"kubernetes.io/projected/ef6dd537-7a92-4d81-8277-de36eafb5dfb-kube-api-access-86q5x\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:15:05.530141 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.530105 2560 generic.go:356] "Generic (PLEG): container finished" podID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerID="510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8" exitCode=0 Sep 16 05:15:05.530569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.530152 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" Sep 16 05:15:05.530569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.530186 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerDied","Data":"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8"} Sep 16 05:15:05.530569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.530241 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv" event={"ID":"ef6dd537-7a92-4d81-8277-de36eafb5dfb","Type":"ContainerDied","Data":"5c538ea2379ee6272085b7d1932d405c75775fbf173171b81e720e2e7d3ff84a"} Sep 16 05:15:05.530569 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.530269 2560 scope.go:117] "RemoveContainer" containerID="510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8" Sep 16 05:15:05.542327 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.542304 2560 scope.go:117] "RemoveContainer" containerID="960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5" Sep 16 05:15:05.555986 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.555956 2560 scope.go:117] "RemoveContainer" containerID="510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8" Sep 16 05:15:05.556395 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:15:05.556370 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8\": container with ID starting with 510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8 not found: ID does not exist" containerID="510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8" Sep 16 05:15:05.556485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.556404 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8"} err="failed to get container status \"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8\": rpc error: code = NotFound desc = could not find container \"510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8\": container with ID starting with 510a4a8f2c93abe2bdda1b06989b04f76297618720266acf7d4b5ee3b870acd8 not found: ID does not exist" Sep 16 05:15:05.556485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.556425 2560 scope.go:117] "RemoveContainer" containerID="960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5" Sep 16 05:15:05.556790 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:15:05.556769 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5\": container with ID starting with 960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5 not found: ID does not exist" containerID="960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5" Sep 16 05:15:05.556863 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.556798 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5"} err="failed to get container status \"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5\": rpc error: code = NotFound desc = could not find container \"960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5\": container with ID starting with 960468cbe08fb7ea0157124c1314b1f28735b80e7c01471e4916820d517911a5 not found: ID does not exist" Sep 16 05:15:05.571976 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.571923 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:15:05.575713 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.575684 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ee7a60-openshift-default-59b68565b8-7rqwv"] Sep 16 05:15:05.671617 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.671577 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" Sep 16 05:15:05.683166 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:05.683142 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" Sep 16 05:15:06.539653 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.539625 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" Sep 16 05:15:06.540119 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.539668 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb" event={"ID":"a1bb853f-deec-451f-a44f-8257f08f9353","Type":"ContainerDied","Data":"ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398"} Sep 16 05:15:06.540119 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.539720 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad83ef0d1adaeb06e1e6616a919ab0726179d681ad743cb782e1a7ae8bc20398" Sep 16 05:15:06.541071 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.541049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" event={"ID":"bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe","Type":"ContainerDied","Data":"b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e"} Sep 16 05:15:06.541071 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.541063 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx" Sep 16 05:15:06.541259 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:06.541086 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b424422686ce4b59548712d86160bdd1c04f9929055dc75f5b4eb87016043b1e" Sep 16 05:15:07.123655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:07.123614 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" path="/var/lib/kubelet/pods/ef6dd537-7a92-4d81-8277-de36eafb5dfb/volumes" Sep 16 05:15:33.896144 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:33.895158 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:43696->10.132.0.85:15021: read: connection reset by peer" start-of-body= Sep 16 05:15:33.896144 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:33.895246 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:43696->10.132.0.85:15021: read: connection reset by peer" Sep 16 05:15:34.737233 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.737202 2560 generic.go:356] "Generic (PLEG): container finished" podID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerID="c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb" exitCode=0 Sep 16 05:15:34.737396 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.737252 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerDied","Data":"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb"} Sep 16 05:15:34.737671 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.737656 2560 scope.go:117] "RemoveContainer" containerID="c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb" Sep 16 05:15:34.738993 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.738963 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:34.739072 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.739032 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:34.739110 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:34.739082 2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 16 05:15:35.746149 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:35.746113 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerStarted","Data":"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab"} Sep 16 05:15:36.095999 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:36.095961 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:36.096272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:36.096246 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:36.096367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:36.096324 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:37.095968 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:37.095930 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:37.096393 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:37.096013 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:38.095855 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:38.095818 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:38.096027 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:38.095876 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:39.096835 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:39.096798 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:39.097260 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:39.096900 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:40.095820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:40.095789 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:40.095989 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:40.095847 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:41.095981 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:41.095948 2560 patch_prober.go:28] interesting pod/e2e-api-b-113278-openshift-default-645647b4cd-h56mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Sep 16 05:15:41.096367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:41.096001 2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Sep 16 05:15:42.099608 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:42.099577 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:42.099985 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:42.099862 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:42.100721 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:42.100705 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:15:52.240161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:52.240077 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb"] Sep 16 05:15:52.245295 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:52.245271 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-75559a-29825595-m5dlb"] Sep 16 05:15:53.122002 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:15:53.121968 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1bb853f-deec-451f-a44f-8257f08f9353" path="/var/lib/kubelet/pods/a1bb853f-deec-451f-a44f-8257f08f9353/volumes" Sep 16 05:16:11.938422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:11.938384 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:16:21.599258 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:21.599225 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:16:21.599770 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:21.599447 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" containerID="cri-o://67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab" gracePeriod=30 Sep 16 05:16:23.053166 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.053125 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:16:23.053760 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.053402 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" podUID="432e564f-441d-4647-a189-56df9c56fb04" containerName="payload-processing" containerID="cri-o://0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76" gracePeriod=30 Sep 16 05:16:23.317587 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.317507 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:16:23.339049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.339017 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume\") pod \"432e564f-441d-4647-a189-56df9c56fb04\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " Sep 16 05:16:23.339225 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.339102 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkcfl\" (UniqueName: \"kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl\") pod \"432e564f-441d-4647-a189-56df9c56fb04\" (UID: \"432e564f-441d-4647-a189-56df9c56fb04\") " Sep 16 05:16:23.339459 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.339431 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "432e564f-441d-4647-a189-56df9c56fb04" (UID: "432e564f-441d-4647-a189-56df9c56fb04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:16:23.341301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.341276 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl" (OuterVolumeSpecName: "kube-api-access-bkcfl") pod "432e564f-441d-4647-a189-56df9c56fb04" (UID: "432e564f-441d-4647-a189-56df9c56fb04"). InnerVolumeSpecName "kube-api-access-bkcfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:23.440809 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.440778 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkcfl\" (UniqueName: \"kubernetes.io/projected/432e564f-441d-4647-a189-56df9c56fb04-kube-api-access-bkcfl\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:23.440809 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:23.440806 2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432e564f-441d-4647-a189-56df9c56fb04-plugins-config-volume\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:24.096595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.096529 2560 generic.go:356] "Generic (PLEG): container finished" podID="432e564f-441d-4647-a189-56df9c56fb04" containerID="0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76" exitCode=0 Sep 16 05:16:24.096595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.096598 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" Sep 16 05:16:24.097189 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.096613 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" event={"ID":"432e564f-441d-4647-a189-56df9c56fb04","Type":"ContainerDied","Data":"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76"} Sep 16 05:16:24.097189 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.096675 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9" event={"ID":"432e564f-441d-4647-a189-56df9c56fb04","Type":"ContainerDied","Data":"0fa90dae4109f528b21bc40799d7b907c959a3a6aecd0b23bcef8d0908a3ab37"} Sep 16 05:16:24.097189 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.096697 2560 scope.go:117] "RemoveContainer" containerID="0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76" Sep 16 05:16:24.110161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.110142 2560 scope.go:117] "RemoveContainer" containerID="0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76" Sep 16 05:16:24.110472 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:24.110447 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76\": container with ID starting with 0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76 not found: ID does not exist" containerID="0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76" Sep 16 05:16:24.110528 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.110479 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76"} err="failed to get container status \"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76\": rpc error: code = NotFound desc = could not find container \"0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76\": container with ID starting with 0f615defbbc7c3305ba3e5521ae324a43387d29a4092fb87cd24c8adc5a12b76 not found: ID does not exist" Sep 16 05:16:24.136526 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.136499 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:16:24.141531 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:24.141508 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-12b6c6-6f99cbcd67-496c9"] Sep 16 05:16:25.121692 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:25.121658 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432e564f-441d-4647-a189-56df9c56fb04" path="/var/lib/kubelet/pods/432e564f-441d-4647-a189-56df9c56fb04/volumes" Sep 16 05:16:26.851789 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.851756 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:16:26.957857 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.957744 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" podUID="67797014-6b4c-4b9b-8b98-0fa55a184c62" containerName="main" containerID="cri-o://68e5d6fad76292e123d320a9aee8d70bb8dbcf9f33254c23b5ff0fdb655dddb7" gracePeriod=45 Sep 16 05:16:26.979391 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979360 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979542 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979403 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979625 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979577 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979625 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979616 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979736 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979627 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:26.979736 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979654 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979736 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979698 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:26.979736 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979707 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979942 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979776 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grjj4\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979942 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979809 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979942 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979899 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy\") pod \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\" (UID: \"2bdd7ae2-70e9-4434-8df1-7d04f67559f8\") " Sep 16 05:16:26.979942 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979927 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:26.980143 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.979967 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:26.980190 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980173 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:16:26.980348 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980325 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:26.980422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980349 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:26.980422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980358 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:26.980422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980367 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:26.980422 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.980375 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:26.981913 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.981888 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:16:26.982003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.981955 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:26.982205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.982182 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4" (OuterVolumeSpecName: "kube-api-access-grjj4") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "kube-api-access-grjj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:26.982205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:26.982192 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bdd7ae2-70e9-4434-8df1-7d04f67559f8" (UID: "2bdd7ae2-70e9-4434-8df1-7d04f67559f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:27.081857 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.081829 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.081857 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.081859 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.082024 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.081873 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.082024 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.081885 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjj4\" (UniqueName: \"kubernetes.io/projected/2bdd7ae2-70e9-4434-8df1-7d04f67559f8-kube-api-access-grjj4\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.120449 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.120414 2560 generic.go:356] "Generic (PLEG): container finished" podID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerID="67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab" exitCode=0 Sep 16 05:16:27.120663 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.120537 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" Sep 16 05:16:27.122244 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.122218 2560 generic.go:356] "Generic (PLEG): container finished" podID="67797014-6b4c-4b9b-8b98-0fa55a184c62" containerID="68e5d6fad76292e123d320a9aee8d70bb8dbcf9f33254c23b5ff0fdb655dddb7" exitCode=0 Sep 16 05:16:27.124971 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.124948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerDied","Data":"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab"} Sep 16 05:16:27.125097 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.124989 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm" event={"ID":"2bdd7ae2-70e9-4434-8df1-7d04f67559f8","Type":"ContainerDied","Data":"574a7e9cdcc297ab093cba0d9d930ce000b9cc480c3109c683de8f547ccb6fbe"} Sep 16 05:16:27.125097 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.125014 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" event={"ID":"67797014-6b4c-4b9b-8b98-0fa55a184c62","Type":"ContainerDied","Data":"68e5d6fad76292e123d320a9aee8d70bb8dbcf9f33254c23b5ff0fdb655dddb7"} Sep 16 05:16:27.125097 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.125042 2560 scope.go:117] "RemoveContainer" containerID="67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab" Sep 16 05:16:27.137990 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.137972 2560 scope.go:117] "RemoveContainer" containerID="c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb" Sep 16 05:16:27.147161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.147143 2560 scope.go:117] "RemoveContainer" containerID="67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab" Sep 16 05:16:27.147541 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:27.147515 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab\": container with ID starting with 67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab not found: ID does not exist" containerID="67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab" Sep 16 05:16:27.147747 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.147565 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab"} err="failed to get container status \"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab\": rpc error: code = NotFound desc = could not find container \"67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab\": container with ID starting with 67a0d9f6788dcf91340e01160b892a6a50959f74e6c3b96ac45a976bd1b7dfab not found: ID does not exist" Sep 16 05:16:27.147747 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.147590 2560 scope.go:117] "RemoveContainer" containerID="c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb" Sep 16 05:16:27.147923 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:27.147904 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb\": container with ID starting with c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb not found: ID does not exist" containerID="c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb" Sep 16 05:16:27.147977 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.147931 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb"} err="failed to get container status \"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb\": rpc error: code = NotFound desc = could not find container \"c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb\": container with ID starting with c31e5c2dd816d52168ce7944854e6cd8c9c6e0bb15b26b5e9a3677b6ba5d2adb not found: ID does not exist" Sep 16 05:16:27.148607 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.148584 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:16:27.157735 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.157711 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-113278-openshift-default-645647b4cd-h56mm"] Sep 16 05:16:27.205604 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.205535 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:16:27.283984 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.283903 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.283984 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.283954 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.284205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.283987 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.284205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284035 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.284205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284076 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.284205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284156 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs\") pod \"67797014-6b4c-4b9b-8b98-0fa55a184c62\" (UID: \"67797014-6b4c-4b9b-8b98-0fa55a184c62\") " Sep 16 05:16:27.284426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284210 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:27.284426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284250 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home" (OuterVolumeSpecName: "home") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:27.284426 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284280 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache" (OuterVolumeSpecName: "model-cache") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:27.284646 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284587 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-tmp-dir\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.284646 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284606 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-model-cache\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.284646 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.284630 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-home\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.286301 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.286270 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v" (OuterVolumeSpecName: "kube-api-access-gqz8v") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "kube-api-access-gqz8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:27.286615 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.286585 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm" (OuterVolumeSpecName: "dshm") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:27.286718 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.286658 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67797014-6b4c-4b9b-8b98-0fa55a184c62" (UID: "67797014-6b4c-4b9b-8b98-0fa55a184c62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:16:27.386012 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.385984 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67797014-6b4c-4b9b-8b98-0fa55a184c62-dshm\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.386012 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.386009 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67797014-6b4c-4b9b-8b98-0fa55a184c62-tls-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:27.386012 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:27.386019 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqz8v\" (UniqueName: \"kubernetes.io/projected/67797014-6b4c-4b9b-8b98-0fa55a184c62-kube-api-access-gqz8v\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:28.132010 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:28.131977 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" Sep 16 05:16:28.132010 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:28.131997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw" event={"ID":"67797014-6b4c-4b9b-8b98-0fa55a184c62","Type":"ContainerDied","Data":"192e37298c39bf46948560e0008c3709cb574a446a9c035a2872512b8ece0df0"} Sep 16 05:16:28.132491 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:28.132046 2560 scope.go:117] "RemoveContainer" containerID="68e5d6fad76292e123d320a9aee8d70bb8dbcf9f33254c23b5ff0fdb655dddb7" Sep 16 05:16:28.154590 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:28.154539 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:16:28.158525 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:28.158506 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-12b6c6/premium-w4-12b6c6-kserve-5d4b89cd86-99cnw"] Sep 16 05:16:29.122056 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:29.122025 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" path="/var/lib/kubelet/pods/2bdd7ae2-70e9-4434-8df1-7d04f67559f8/volumes" Sep 16 05:16:29.122640 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:29.122617 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67797014-6b4c-4b9b-8b98-0fa55a184c62" path="/var/lib/kubelet/pods/67797014-6b4c-4b9b-8b98-0fa55a184c62/volumes" Sep 16 05:16:33.022013 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.021978 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:16:33.022412 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.022231 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" podUID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" containerName="maas-api" containerID="cri-o://0d7563df0cefa16bf9bcfed1c3d1c8619ed88c39c68fd2a482c6cf163d128cda" gracePeriod=30 Sep 16 05:16:33.168258 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.168227 2560 generic.go:356] "Generic (PLEG): container finished" podID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" containerID="0d7563df0cefa16bf9bcfed1c3d1c8619ed88c39c68fd2a482c6cf163d128cda" exitCode=0 Sep 16 05:16:33.168411 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.168289 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" event={"ID":"d6b3033b-3c96-4e9b-bb01-fe9284869f58","Type":"ContainerDied","Data":"0d7563df0cefa16bf9bcfed1c3d1c8619ed88c39c68fd2a482c6cf163d128cda"} Sep 16 05:16:33.290731 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.290709 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:16:33.342864 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.342823 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") pod \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " Sep 16 05:16:33.343063 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.342889 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs\") pod \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " Sep 16 05:16:33.343063 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.342972 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca\") pod \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " Sep 16 05:16:33.343201 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.343123 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djw9r\" (UniqueName: \"kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r\") pod \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\" (UID: \"d6b3033b-3c96-4e9b-bb01-fe9284869f58\") " Sep 16 05:16:33.343347 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.343319 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6b3033b-3c96-4e9b-bb01-fe9284869f58" (UID: "d6b3033b-3c96-4e9b-bb01-fe9284869f58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:16:33.343599 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.343581 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b3033b-3c96-4e9b-bb01-fe9284869f58-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:33.345272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.345243 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6b3033b-3c96-4e9b-bb01-fe9284869f58" (UID: "d6b3033b-3c96-4e9b-bb01-fe9284869f58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:16:33.345367 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.345274 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r" (OuterVolumeSpecName: "kube-api-access-djw9r") pod "d6b3033b-3c96-4e9b-bb01-fe9284869f58" (UID: "d6b3033b-3c96-4e9b-bb01-fe9284869f58"). InnerVolumeSpecName "kube-api-access-djw9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:33.345520 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.345498 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d6b3033b-3c96-4e9b-bb01-fe9284869f58" (UID: "d6b3033b-3c96-4e9b-bb01-fe9284869f58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:16:33.444317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.444275 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djw9r\" (UniqueName: \"kubernetes.io/projected/d6b3033b-3c96-4e9b-bb01-fe9284869f58-kube-api-access-djw9r\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:33.444317 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.444319 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:33.444525 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:33.444334 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6b3033b-3c96-4e9b-bb01-fe9284869f58-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:34.124072 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124036 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7847-39a6e0"] Sep 16 05:16:34.124440 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124428 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124442 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124450 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe" containerName="cleanup" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124455 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe" containerName="cleanup" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124464 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67797014-6b4c-4b9b-8b98-0fa55a184c62" containerName="main" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124470 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="67797014-6b4c-4b9b-8b98-0fa55a184c62" containerName="main" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124479 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" containerName="maas-api" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124484 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" containerName="maas-api" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124492 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1bb853f-deec-451f-a44f-8257f08f9353" containerName="cleanup" Sep 16 05:16:34.124497 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124497 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1bb853f-deec-451f-a44f-8257f08f9353" containerName="cleanup" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124505 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432e564f-441d-4647-a189-56df9c56fb04" containerName="payload-processing" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124511 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="432e564f-441d-4647-a189-56df9c56fb04" containerName="payload-processing" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124519 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124527 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124537 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124543 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124563 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124568 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124634 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="432e564f-441d-4647-a189-56df9c56fb04" containerName="payload-processing" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124642 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="67797014-6b4c-4b9b-8b98-0fa55a184c62" containerName="main" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124651 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124656 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6dd537-7a92-4d81-8277-de36eafb5dfb" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124661 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe" containerName="cleanup" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124668 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1bb853f-deec-451f-a44f-8257f08f9353" containerName="cleanup" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124673 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124679 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" containerName="maas-api" Sep 16 05:16:34.124803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.124684 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bdd7ae2-70e9-4434-8df1-7d04f67559f8" containerName="istio-proxy" Sep 16 05:16:34.127897 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.127879 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:34.131695 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.131661 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fr2c9\"" Sep 16 05:16:34.133174 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.133148 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-39a6e0"] Sep 16 05:16:34.176399 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.176373 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" Sep 16 05:16:34.176523 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.176408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc" event={"ID":"d6b3033b-3c96-4e9b-bb01-fe9284869f58","Type":"ContainerDied","Data":"ecf395ef47f22d7115219c40425d0fd10252dc39c65bc6f10fcd9f8910dd791d"} Sep 16 05:16:34.176523 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.176463 2560 scope.go:117] "RemoveContainer" containerID="0d7563df0cefa16bf9bcfed1c3d1c8619ed88c39c68fd2a482c6cf163d128cda" Sep 16 05:16:34.198310 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.198283 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:16:34.205451 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.205428 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ad06de-55cfbfb4b5-pjjcc"] Sep 16 05:16:34.252812 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.252792 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbx77\" (UniqueName: \"kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77\") pod \"test-curl-7847-39a6e0\" (UID: \"4ffd1ef6-787c-4e47-be82-6878eaa2c169\") " pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:34.353828 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.353796 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbx77\" (UniqueName: \"kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77\") pod \"test-curl-7847-39a6e0\" (UID: \"4ffd1ef6-787c-4e47-be82-6878eaa2c169\") " pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:34.363580 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.363528 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbx77\" (UniqueName: \"kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77\") pod \"test-curl-7847-39a6e0\" (UID: \"4ffd1ef6-787c-4e47-be82-6878eaa2c169\") " pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:34.439486 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.439371 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:34.715207 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:34.715178 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-39a6e0"] Sep 16 05:16:34.716443 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:16:34.716416 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ffd1ef6_787c_4e47_be82_6878eaa2c169.slice/crio-37f43c10f8c292ca5b8fd938ce35804fe00ab43c36526f6840088e4ed5002c47 WatchSource:0}: Error finding container 37f43c10f8c292ca5b8fd938ce35804fe00ab43c36526f6840088e4ed5002c47: Status 404 returned error can't find the container with id 37f43c10f8c292ca5b8fd938ce35804fe00ab43c36526f6840088e4ed5002c47 Sep 16 05:16:35.122296 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.122265 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b3033b-3c96-4e9b-bb01-fe9284869f58" path="/var/lib/kubelet/pods/d6b3033b-3c96-4e9b-bb01-fe9284869f58/volumes" Sep 16 05:16:35.184159 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.184129 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-39a6e0" event={"ID":"4ffd1ef6-787c-4e47-be82-6878eaa2c169","Type":"ContainerStarted","Data":"61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0"} Sep 16 05:16:35.184159 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.184164 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-39a6e0" event={"ID":"4ffd1ef6-787c-4e47-be82-6878eaa2c169","Type":"ContainerStarted","Data":"37f43c10f8c292ca5b8fd938ce35804fe00ab43c36526f6840088e4ed5002c47"} Sep 16 05:16:35.215084 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.215042 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7847-39a6e0" podStartSLOduration=1.215029219 podStartE2EDuration="1.215029219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:16:35.212886454 +0000 UTC m=+1660.726437987" watchObservedRunningTime="2026-09-16 05:16:35.215029219 +0000 UTC m=+1660.728580751" Sep 16 05:16:35.841345 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.841314 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7847-39a6e0"] Sep 16 05:16:35.847567 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:35.847527 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7847-39a6e0"] Sep 16 05:16:36.190717 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:36.190642 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7847-39a6e0" container="test-curl-7847-39a6e0" Sep 16 05:16:37.196406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:37.196367 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7847-39a6e0" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" containerName="test-curl-7847-39a6e0" containerID="cri-o://61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0" gracePeriod=2 Sep 16 05:16:39.378803 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.378778 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:39.508033 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.507965 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbx77\" (UniqueName: \"kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77\") pod \"4ffd1ef6-787c-4e47-be82-6878eaa2c169\" (UID: \"4ffd1ef6-787c-4e47-be82-6878eaa2c169\") " Sep 16 05:16:39.509994 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.509969 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77" (OuterVolumeSpecName: "kube-api-access-dbx77") pod "4ffd1ef6-787c-4e47-be82-6878eaa2c169" (UID: "4ffd1ef6-787c-4e47-be82-6878eaa2c169"). InnerVolumeSpecName "kube-api-access-dbx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:39.608773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.608733 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbx77\" (UniqueName: \"kubernetes.io/projected/4ffd1ef6-787c-4e47-be82-6878eaa2c169-kube-api-access-dbx77\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:39.783693 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.783614 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:16:39.783843 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:39.783817 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" containerID="cri-o://56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196" gracePeriod=30 Sep 16 05:16:40.217502 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:40.217410 2560 generic.go:356] "Generic (PLEG): container finished" podID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" containerID="61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0" exitCode=137 Sep 16 05:16:40.217502 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:40.217473 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:40.217766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:40.217536 2560 scope.go:117] "RemoveContainer" containerID="61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0" Sep 16 05:16:40.230332 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:40.230300 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7847-39a6e0\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" pod="opendatahub/test-curl-7847-39a6e0" Sep 16 05:16:40.231067 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:40.231035 2560 scope.go:117] "RemoveContainer" containerID="61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0" Sep 16 05:16:40.231490 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:40.231455 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0\": container with ID starting with 61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0 not found: ID does not exist" containerID="61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0" Sep 16 05:16:40.231601 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:40.231495 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0"} err="failed to get container status \"61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0\": rpc error: code = NotFound desc = could not find container \"61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0\": container with ID starting with 61423002d049b4803521ba53b3f5cec84ad43238b0e3fadc8ef3e1fcba0bf3b0 not found: ID does not exist" Sep 16 05:16:41.121902 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:41.121861 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" path="/var/lib/kubelet/pods/4ffd1ef6-787c-4e47-be82-6878eaa2c169/volumes" Sep 16 05:16:42.246939 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:42.246906 2560 patch_prober.go:28] interesting pod/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 05:16:42.247339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:42.246970 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 05:16:45.042043 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.042015 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:16:45.165381 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165350 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165391 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165431 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165456 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165501 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165643 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165688 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4qst\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165702 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165745 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data" (OuterVolumeSpecName: "istio-data") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165785 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.165813 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165795 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:45.166129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165819 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token\") pod \"8950fe98-8721-4f3d-9f26-d00b0e727938\" (UID: \"8950fe98-8721-4f3d-9f26-d00b0e727938\") " Sep 16 05:16:45.166129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.165870 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:16:45.166129 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166008 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:45.166271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166144 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.166271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166164 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.166271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166179 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.166271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166192 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.166271 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.166204 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8950fe98-8721-4f3d-9f26-d00b0e727938-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.167874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.167852 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst" (OuterVolumeSpecName: "kube-api-access-s4qst") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "kube-api-access-s4qst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:45.167968 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.167873 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:16:45.168164 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.168144 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:16:45.168225 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.168145 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token" (OuterVolumeSpecName: "istio-token") pod "8950fe98-8721-4f3d-9f26-d00b0e727938" (UID: "8950fe98-8721-4f3d-9f26-d00b0e727938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:16:45.251139 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.251113 2560 generic.go:356] "Generic (PLEG): container finished" podID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerID="56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196" exitCode=0 Sep 16 05:16:45.251266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.251157 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" Sep 16 05:16:45.251266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.251199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerDied","Data":"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196"} Sep 16 05:16:45.251266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.251253 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g" event={"ID":"8950fe98-8721-4f3d-9f26-d00b0e727938","Type":"ContainerDied","Data":"9d91a32ac23ab6131e775f9460eaf19ca86acea8c6b937dd97a410f69dc0baab"} Sep 16 05:16:45.251373 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.251285 2560 scope.go:117] "RemoveContainer" containerID="56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196" Sep 16 05:16:45.261152 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.261133 2560 scope.go:117] "RemoveContainer" containerID="18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564" Sep 16 05:16:45.267609 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.267582 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.267722 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.267609 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.267722 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.267626 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4qst\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-kube-api-access-s4qst\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.267722 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.267642 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8950fe98-8721-4f3d-9f26-d00b0e727938-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:16:45.269676 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.269661 2560 scope.go:117] "RemoveContainer" containerID="56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196" Sep 16 05:16:45.269939 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:45.269912 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196\": container with ID starting with 56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196 not found: ID does not exist" containerID="56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196" Sep 16 05:16:45.270049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.269949 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196"} err="failed to get container status \"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196\": rpc error: code = NotFound desc = could not find container \"56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196\": container with ID starting with 56aee043d360411a5273a8e201273e0a981d2c8bb7b906d13015c6b6ab311196 not found: ID does not exist" Sep 16 05:16:45.270049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.269974 2560 scope.go:117] "RemoveContainer" containerID="18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564" Sep 16 05:16:45.270332 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:16:45.270311 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564\": container with ID starting with 18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564 not found: ID does not exist" containerID="18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564" Sep 16 05:16:45.270332 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.270337 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564"} err="failed to get container status \"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564\": rpc error: code = NotFound desc = could not find container \"18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564\": container with ID starting with 18b65c8b85c7b2d7befe0afe91dcb98c9fea46ebe5bd915fee1a956d5f3de564 not found: ID does not exist" Sep 16 05:16:45.273629 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.273607 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:16:45.278139 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:45.278118 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-ad06de-openshift-default-58484fd578-hvr4g"] Sep 16 05:16:47.121556 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:16:47.121513 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" path="/var/lib/kubelet/pods/8950fe98-8721-4f3d-9f26-d00b0e727938/volumes" Sep 16 05:18:30.464435 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464396 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7847-913ec2"] Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464855 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464868 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464885 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464894 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464931 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" containerName="test-curl-7847-39a6e0" Sep 16 05:18:30.464937 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464938 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" containerName="test-curl-7847-39a6e0" Sep 16 05:18:30.465135 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.464998 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:30.465135 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.465010 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffd1ef6-787c-4e47-be82-6878eaa2c169" containerName="test-curl-7847-39a6e0" Sep 16 05:18:30.468146 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.468120 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:30.470893 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.470864 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fr2c9\"" Sep 16 05:18:30.471892 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.471869 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-913ec2"] Sep 16 05:18:30.585938 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.585905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9n9x\" (UniqueName: \"kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x\") pod \"test-curl-7847-913ec2\" (UID: \"c6e93c08-f07c-4612-8378-34ee8fc37de3\") " pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:30.686617 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.686582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9n9x\" (UniqueName: \"kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x\") pod \"test-curl-7847-913ec2\" (UID: \"c6e93c08-f07c-4612-8378-34ee8fc37de3\") " pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:30.695661 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.695638 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9n9x\" (UniqueName: \"kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x\") pod \"test-curl-7847-913ec2\" (UID: \"c6e93c08-f07c-4612-8378-34ee8fc37de3\") " pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:30.779828 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:30.779751 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:31.057019 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:31.056992 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-913ec2"] Sep 16 05:18:31.059081 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:18:31.059054 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e93c08_f07c_4612_8378_34ee8fc37de3.slice/crio-270cff3f0c775822aacc7e64561c66773bb4d3adf715bdfccc018676bdb0860d WatchSource:0}: Error finding container 270cff3f0c775822aacc7e64561c66773bb4d3adf715bdfccc018676bdb0860d: Status 404 returned error can't find the container with id 270cff3f0c775822aacc7e64561c66773bb4d3adf715bdfccc018676bdb0860d Sep 16 05:18:31.886134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:31.886097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-913ec2" event={"ID":"c6e93c08-f07c-4612-8378-34ee8fc37de3","Type":"ContainerStarted","Data":"e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27"} Sep 16 05:18:31.886134 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:31.886137 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-913ec2" event={"ID":"c6e93c08-f07c-4612-8378-34ee8fc37de3","Type":"ContainerStarted","Data":"270cff3f0c775822aacc7e64561c66773bb4d3adf715bdfccc018676bdb0860d"} Sep 16 05:18:31.902321 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:31.902272 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7847-913ec2" podStartSLOduration=1.902244097 podStartE2EDuration="1.902244097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:18:31.901698021 +0000 UTC m=+1777.415249558" watchObservedRunningTime="2026-09-16 05:18:31.902244097 +0000 UTC m=+1777.415795631" Sep 16 05:18:32.488097 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.488063 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7847-913ec2"] Sep 16 05:18:32.492236 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.492204 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7847-913ec2"] Sep 16 05:18:32.610156 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.610122 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7847-4295a6"] Sep 16 05:18:32.610595 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.610579 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" containerName="test-curl-7847-913ec2" Sep 16 05:18:32.610647 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.610597 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" containerName="test-curl-7847-913ec2" Sep 16 05:18:32.610687 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.610669 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8950fe98-8721-4f3d-9f26-d00b0e727938" containerName="istio-proxy" Sep 16 05:18:32.610687 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.610681 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" containerName="test-curl-7847-913ec2" Sep 16 05:18:32.613758 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.613743 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:32.628745 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.628719 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-4295a6"] Sep 16 05:18:32.709111 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.709074 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntt4\" (UniqueName: \"kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4\") pod \"test-curl-7847-4295a6\" (UID: \"31ef8c75-b95e-4758-a21c-0f0e4f55b954\") " pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:32.810401 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.810362 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntt4\" (UniqueName: \"kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4\") pod \"test-curl-7847-4295a6\" (UID: \"31ef8c75-b95e-4758-a21c-0f0e4f55b954\") " pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:32.819576 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.819534 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntt4\" (UniqueName: \"kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4\") pod \"test-curl-7847-4295a6\" (UID: \"31ef8c75-b95e-4758-a21c-0f0e4f55b954\") " pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:32.892338 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:32.892257 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7847-913ec2" container="test-curl-7847-913ec2" Sep 16 05:18:32.923137 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:32.923105 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:33.196857 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:33.196831 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7847-4295a6"] Sep 16 05:18:33.198074 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:18:33.198052 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ef8c75_b95e_4758_a21c_0f0e4f55b954.slice/crio-9b959704c931a8a59e5537ae6dc8012e8ba4f2daf37585ee96ee5c06328de07e WatchSource:0}: Error finding container 9b959704c931a8a59e5537ae6dc8012e8ba4f2daf37585ee96ee5c06328de07e: Status 404 returned error can't find the container with id 9b959704c931a8a59e5537ae6dc8012e8ba4f2daf37585ee96ee5c06328de07e Sep 16 05:18:33.899621 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:33.899578 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-4295a6" event={"ID":"31ef8c75-b95e-4758-a21c-0f0e4f55b954","Type":"ContainerStarted","Data":"61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b"} Sep 16 05:18:33.899621 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:33.899623 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7847-4295a6" event={"ID":"31ef8c75-b95e-4758-a21c-0f0e4f55b954","Type":"ContainerStarted","Data":"9b959704c931a8a59e5537ae6dc8012e8ba4f2daf37585ee96ee5c06328de07e"} Sep 16 05:18:33.900120 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:33.899713 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7847-913ec2" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" containerName="test-curl-7847-913ec2" containerID="cri-o://e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27" gracePeriod=2 Sep 16 05:18:33.915373 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:33.915324 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7847-4295a6" podStartSLOduration=1.915310055 podStartE2EDuration="1.915310055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:18:33.913505555 +0000 UTC m=+1779.427057101" watchObservedRunningTime="2026-09-16 05:18:33.915310055 +0000 UTC m=+1779.428861590" Sep 16 05:18:34.494486 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:34.494455 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7847-4295a6"] Sep 16 05:18:34.498964 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:34.498937 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7847-4295a6"] Sep 16 05:18:34.905156 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:34.905126 2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7847-4295a6" container="test-curl-7847-4295a6" Sep 16 05:18:35.910960 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:35.910924 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7847-4295a6" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" containerName="test-curl-7847-4295a6" containerID="cri-o://61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b" gracePeriod=2 Sep 16 05:18:36.043521 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.043494 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:36.142230 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.142198 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9n9x\" (UniqueName: \"kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x\") pod \"c6e93c08-f07c-4612-8378-34ee8fc37de3\" (UID: \"c6e93c08-f07c-4612-8378-34ee8fc37de3\") " Sep 16 05:18:36.144286 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.144264 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x" (OuterVolumeSpecName: "kube-api-access-t9n9x") pod "c6e93c08-f07c-4612-8378-34ee8fc37de3" (UID: "c6e93c08-f07c-4612-8378-34ee8fc37de3"). InnerVolumeSpecName "kube-api-access-t9n9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:18:36.243618 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.243507 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9n9x\" (UniqueName: \"kubernetes.io/projected/c6e93c08-f07c-4612-8378-34ee8fc37de3-kube-api-access-t9n9x\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:18:36.917948 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.917910 2560 generic.go:356] "Generic (PLEG): container finished" podID="c6e93c08-f07c-4612-8378-34ee8fc37de3" containerID="e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27" exitCode=137 Sep 16 05:18:36.918377 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.917959 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:36.918377 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.918020 2560 scope.go:117] "RemoveContainer" containerID="e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27" Sep 16 05:18:36.927164 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.927147 2560 scope.go:117] "RemoveContainer" containerID="e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27" Sep 16 05:18:36.927402 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:36.927378 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27\": container with ID starting with e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27 not found: ID does not exist" containerID="e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27" Sep 16 05:18:36.927469 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:36.927413 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27"} err="failed to get container status \"e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27\": rpc error: code = NotFound desc = could not find container \"e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27\": container with ID starting with e3daca4bb042da77fd1994c4c6073f5046674c1051eb571aa2d34c9eaba9ce27 not found: ID does not exist" Sep 16 05:18:36.928502 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:36.928478 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7847-913ec2\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" pod="opendatahub/test-curl-7847-913ec2" Sep 16 05:18:37.121747 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:37.121714 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e93c08-f07c-4612-8378-34ee8fc37de3" path="/var/lib/kubelet/pods/c6e93c08-f07c-4612-8378-34ee8fc37de3/volumes" Sep 16 05:18:38.109338 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.109312 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:38.265972 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.265881 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ntt4\" (UniqueName: \"kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4\") pod \"31ef8c75-b95e-4758-a21c-0f0e4f55b954\" (UID: \"31ef8c75-b95e-4758-a21c-0f0e4f55b954\") " Sep 16 05:18:38.268013 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.267986 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4" (OuterVolumeSpecName: "kube-api-access-4ntt4") pod "31ef8c75-b95e-4758-a21c-0f0e4f55b954" (UID: "31ef8c75-b95e-4758-a21c-0f0e4f55b954"). InnerVolumeSpecName "kube-api-access-4ntt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:18:38.367658 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.367618 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntt4\" (UniqueName: \"kubernetes.io/projected/31ef8c75-b95e-4758-a21c-0f0e4f55b954-kube-api-access-4ntt4\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:18:38.933905 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.933868 2560 generic.go:356] "Generic (PLEG): container finished" podID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" containerID="61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b" exitCode=137 Sep 16 05:18:38.934068 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.933912 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:38.934068 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.933976 2560 scope.go:117] "RemoveContainer" containerID="61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b" Sep 16 05:18:38.943209 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.943191 2560 scope.go:117] "RemoveContainer" containerID="61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b" Sep 16 05:18:38.943493 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:38.943473 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b\": container with ID starting with 61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b not found: ID does not exist" containerID="61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b" Sep 16 05:18:38.943543 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:38.943502 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b"} err="failed to get container status \"61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b\": rpc error: code = NotFound desc = could not find container \"61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b\": container with ID starting with 61b676cc9d1b2956b7536d1a158b007187bf72fccefe05b5faff6ed1a4b8df9b not found: ID does not exist" Sep 16 05:18:38.944149 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:18:38.944132 2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7847-4295a6\" is forbidden: User \"system:node:ip-10-0-142-144.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-144.ec2.internal' and this object" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" pod="opendatahub/test-curl-7847-4295a6" Sep 16 05:18:39.121884 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:39.121853 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" path="/var/lib/kubelet/pods/31ef8c75-b95e-4758-a21c-0f0e4f55b954/volumes" Sep 16 05:18:55.173340 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:55.173297 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:18:55.176420 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:55.176398 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:18:55.176702 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:55.176681 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:18:55.179688 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:18:55.179672 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:20:32.602498 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.602419 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:20:32.603135 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.602920 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" containerName="test-curl-7847-4295a6" Sep 16 05:20:32.603135 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.602934 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" containerName="test-curl-7847-4295a6" Sep 16 05:20:32.603135 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.602998 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ef8c75-b95e-4758-a21c-0f0e4f55b954" containerName="test-curl-7847-4295a6" Sep 16 05:20:32.606153 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.606134 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.608919 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.608896 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-afd377-serving-cert\"" Sep 16 05:20:32.616182 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.616155 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:20:32.670949 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.670913 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.671118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.670977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhg5\" (UniqueName: \"kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.671118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.671015 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.671118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.671039 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.771814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.771778 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhg5\" (UniqueName: \"kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.771814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.771817 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.772074 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.771845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.772074 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.771896 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.772536 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.772513 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.774325 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.774305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.774420 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.774351 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.781468 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.781442 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhg5\" (UniqueName: \"kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5\") pod \"maas-api-e2e-ipp-b-afd377-f95696ccd-764wg\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:32.918164 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:32.918056 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:33.202568 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:20:33.202508 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7449ecf0_7357_47ff_b7a5_80a2ae2ecb8b.slice/crio-23e4ff4c73abeb3d3dc059c2ba4f960bbe80408915029c4f3b3accca8d343750 WatchSource:0}: Error finding container 23e4ff4c73abeb3d3dc059c2ba4f960bbe80408915029c4f3b3accca8d343750: Status 404 returned error can't find the container with id 23e4ff4c73abeb3d3dc059c2ba4f960bbe80408915029c4f3b3accca8d343750 Sep 16 05:20:33.203109 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:33.203082 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:20:33.204299 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:33.204278 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 05:20:33.611921 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:33.611887 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" event={"ID":"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b","Type":"ContainerStarted","Data":"23e4ff4c73abeb3d3dc059c2ba4f960bbe80408915029c4f3b3accca8d343750"} Sep 16 05:20:34.621463 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:34.621417 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" event={"ID":"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b","Type":"ContainerStarted","Data":"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f"} Sep 16 05:20:34.623860 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:34.621638 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:20:34.641661 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:34.641605 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" podStartSLOduration=2.150864394 podStartE2EDuration="2.641582328s" totalImagesPullingTime="490.717934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:20:34.638267611 +0000 UTC m=+1900.151819156" watchObservedRunningTime="2026-09-16 05:20:34.641582328 +0000 UTC m=+1900.155133862" Sep 16 05:20:40.633825 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:20:40.633793 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:21:42.617110 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.617077 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:21:42.621132 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.621107 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.623745 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.623721 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-f644d7-serving-cert\"" Sep 16 05:21:42.633078 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.633057 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:21:42.677276 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.677245 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.677442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.677303 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.677442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.677334 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.677442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.677398 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45z2w\" (UniqueName: \"kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.778170 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.778134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.778360 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.778180 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.778360 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.778230 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45z2w\" (UniqueName: \"kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.778360 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.778275 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.778360 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:21:42.778340 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-serving-cert: secret "maas-api-e2e-ipp-cleanup-f644d7-serving-cert" not found Sep 16 05:21:42.778638 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:21:42.778403 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls podName:07bb45fa-4854-4d27-a922-107003a1e30b nodeName:}" failed. No retries permitted until 2026-09-16 05:21:43.278389032 +0000 UTC m=+1968.791940544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b") : secret "maas-api-e2e-ipp-cleanup-f644d7-serving-cert" not found Sep 16 05:21:42.778904 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.778889 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.780540 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.780516 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:42.794052 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:42.794026 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45z2w\" (UniqueName: \"kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:43.284141 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:43.284101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:43.284361 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:21:43.284293 2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-serving-cert: secret "maas-api-e2e-ipp-cleanup-f644d7-serving-cert" not found Sep 16 05:21:43.284476 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:21:43.284364 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls podName:07bb45fa-4854-4d27-a922-107003a1e30b nodeName:}" failed. No retries permitted until 2026-09-16 05:21:44.28434297 +0000 UTC m=+1969.797894500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b") : secret "maas-api-e2e-ipp-cleanup-f644d7-serving-cert" not found Sep 16 05:21:44.294414 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:44.294373 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:44.296859 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:44.296838 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:44.434114 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:44.434053 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:44.708615 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:44.708584 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:21:44.709670 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:21:44.709644 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bb45fa_4854_4d27_a922_107003a1e30b.slice/crio-47a999a4848941fb1f3736e75d464f2fd086a4604d1b24f3f4a8a15edd376d1d WatchSource:0}: Error finding container 47a999a4848941fb1f3736e75d464f2fd086a4604d1b24f3f4a8a15edd376d1d: Status 404 returned error can't find the container with id 47a999a4848941fb1f3736e75d464f2fd086a4604d1b24f3f4a8a15edd376d1d Sep 16 05:21:45.043618 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:45.043516 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" event={"ID":"07bb45fa-4854-4d27-a922-107003a1e30b","Type":"ContainerStarted","Data":"47a999a4848941fb1f3736e75d464f2fd086a4604d1b24f3f4a8a15edd376d1d"} Sep 16 05:21:46.051969 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:46.051932 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" event={"ID":"07bb45fa-4854-4d27-a922-107003a1e30b","Type":"ContainerStarted","Data":"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796"} Sep 16 05:21:46.052342 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:46.052157 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:46.069861 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:46.069820 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" podStartSLOduration=3.60411792 podStartE2EDuration="4.069806259s" totalImagesPullingTime="465.688339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:21:46.068174701 +0000 UTC m=+1971.581726236" watchObservedRunningTime="2026-09-16 05:21:46.069806259 +0000 UTC m=+1971.583357793" Sep 16 05:21:52.064873 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:52.064789 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:21:55.344209 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:21:55.344176 2560 scope.go:117] "RemoveContainer" containerID="6802e3489f67752b36abe514616101b4741291c33c099a1ddf6a4d92d2c8aaa0" Sep 16 05:22:11.702997 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:11.702962 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:22:11.703602 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:11.703239 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" podUID="07bb45fa-4854-4d27-a922-107003a1e30b" containerName="maas-api" containerID="cri-o://ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796" gracePeriod=30 Sep 16 05:22:11.957944 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:11.957877 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:22:12.064576 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.064512 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs\") pod \"07bb45fa-4854-4d27-a922-107003a1e30b\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " Sep 16 05:22:12.064766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.064699 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") pod \"07bb45fa-4854-4d27-a922-107003a1e30b\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " Sep 16 05:22:12.064766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.064752 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45z2w\" (UniqueName: \"kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w\") pod \"07bb45fa-4854-4d27-a922-107003a1e30b\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " Sep 16 05:22:12.064874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.064786 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca\") pod \"07bb45fa-4854-4d27-a922-107003a1e30b\" (UID: \"07bb45fa-4854-4d27-a922-107003a1e30b\") " Sep 16 05:22:12.065211 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.065177 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07bb45fa-4854-4d27-a922-107003a1e30b" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:22:12.066771 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.066744 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07bb45fa-4854-4d27-a922-107003a1e30b" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:22:12.066989 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.066965 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07bb45fa-4854-4d27-a922-107003a1e30b" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:22:12.067091 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.067067 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w" (OuterVolumeSpecName: "kube-api-access-45z2w") pod "07bb45fa-4854-4d27-a922-107003a1e30b" (UID: "07bb45fa-4854-4d27-a922-107003a1e30b"). InnerVolumeSpecName "kube-api-access-45z2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:22:12.165918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.165879 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45z2w\" (UniqueName: \"kubernetes.io/projected/07bb45fa-4854-4d27-a922-107003a1e30b-kube-api-access-45z2w\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:22:12.165918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.165910 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07bb45fa-4854-4d27-a922-107003a1e30b-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:22:12.165918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.165921 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:22:12.166191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.165936 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07bb45fa-4854-4d27-a922-107003a1e30b-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:22:12.206257 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.206223 2560 generic.go:356] "Generic (PLEG): container finished" podID="07bb45fa-4854-4d27-a922-107003a1e30b" containerID="ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796" exitCode=0 Sep 16 05:22:12.206406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.206265 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" Sep 16 05:22:12.206406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.206307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" event={"ID":"07bb45fa-4854-4d27-a922-107003a1e30b","Type":"ContainerDied","Data":"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796"} Sep 16 05:22:12.206406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.206356 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb" event={"ID":"07bb45fa-4854-4d27-a922-107003a1e30b","Type":"ContainerDied","Data":"47a999a4848941fb1f3736e75d464f2fd086a4604d1b24f3f4a8a15edd376d1d"} Sep 16 05:22:12.206406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.206382 2560 scope.go:117] "RemoveContainer" containerID="ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796" Sep 16 05:22:12.216353 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.216334 2560 scope.go:117] "RemoveContainer" containerID="ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796" Sep 16 05:22:12.216631 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:22:12.216611 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796\": container with ID starting with ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796 not found: ID does not exist" containerID="ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796" Sep 16 05:22:12.216712 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.216642 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796"} err="failed to get container status \"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796\": rpc error: code = NotFound desc = could not find container \"ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796\": container with ID starting with ab2fa5ddc6b55a6d3be4d938ba3089375c7a3ed0bc7ffbce3eb753deadaf5796 not found: ID does not exist" Sep 16 05:22:12.228740 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.228711 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:22:12.232475 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:12.232454 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-f644d7-568bbc9b59-np5hb"] Sep 16 05:22:13.122083 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:22:13.122053 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07bb45fa-4854-4d27-a922-107003a1e30b" path="/var/lib/kubelet/pods/07bb45fa-4854-4d27-a922-107003a1e30b/volumes" Sep 16 05:23:37.404249 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.404219 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:23:37.404703 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.404434 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" podUID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" containerName="maas-api" containerID="cri-o://590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f" gracePeriod=30 Sep 16 05:23:37.660259 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.660184 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:23:37.711929 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.711888 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca\") pod \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " Sep 16 05:23:37.712106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.712034 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwhg5\" (UniqueName: \"kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5\") pod \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " Sep 16 05:23:37.712106 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.712092 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs\") pod \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " Sep 16 05:23:37.712224 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.712130 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls\") pod \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\" (UID: \"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b\") " Sep 16 05:23:37.712436 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.712403 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" (UID: "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:23:37.712689 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.712666 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:23:37.714247 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.714215 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5" (OuterVolumeSpecName: "kube-api-access-dwhg5") pod "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" (UID: "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b"). InnerVolumeSpecName "kube-api-access-dwhg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:23:37.714685 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.714662 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" (UID: "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:23:37.714901 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.714878 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" (UID: "7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:23:37.728049 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.727939 2560 generic.go:356] "Generic (PLEG): container finished" podID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" containerID="590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f" exitCode=0 Sep 16 05:23:37.728158 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.728057 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" Sep 16 05:23:37.728158 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.728087 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" event={"ID":"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b","Type":"ContainerDied","Data":"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f"} Sep 16 05:23:37.728158 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.728143 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg" event={"ID":"7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b","Type":"ContainerDied","Data":"23e4ff4c73abeb3d3dc059c2ba4f960bbe80408915029c4f3b3accca8d343750"} Sep 16 05:23:37.728343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.728174 2560 scope.go:117] "RemoveContainer" containerID="590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f" Sep 16 05:23:37.740149 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.740126 2560 scope.go:117] "RemoveContainer" containerID="590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f" Sep 16 05:23:37.740483 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:23:37.740464 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f\": container with ID starting with 590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f not found: ID does not exist" containerID="590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f" Sep 16 05:23:37.740579 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.740491 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f"} err="failed to get container status \"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f\": rpc error: code = NotFound desc = could not find container \"590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f\": container with ID starting with 590acf47b05c9de6da795f42fdbc8c7e862239b0924352e4a479173eb170d18f not found: ID does not exist" Sep 16 05:23:37.751991 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.751968 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:23:37.755652 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.755631 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-afd377-f95696ccd-764wg"] Sep 16 05:23:37.813820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.813784 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:23:37.813820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.813813 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:23:37.813820 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:37.813823 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhg5\" (UniqueName: \"kubernetes.io/projected/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b-kube-api-access-dwhg5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:23:39.122036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:39.121993 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" path="/var/lib/kubelet/pods/7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b/volumes" Sep 16 05:23:55.216413 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:55.216371 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:23:55.219575 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:55.219535 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:23:55.221671 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:55.221647 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:23:55.225067 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:23:55.225049 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:24:01.829932 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:01.829898 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:24:01.830392 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:01.830117 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" containerID="cri-o://4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269" gracePeriod=30 Sep 16 05:24:07.088477 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.088451 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:24:07.094865 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094807 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.094865 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094842 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094871 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094908 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094942 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095040 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.094982 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdd44\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095069 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:07.095242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095115 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095210 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095223 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:07.095242 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095241 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert\") pod \"5a00bc64-dbf8-497a-8f86-2650ceff85de\" (UID: \"5a00bc64-dbf8-497a-8f86-2650ceff85de\") " Sep 16 05:24:07.095586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095236 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:07.095586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095446 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:07.095733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095609 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:24:07.095733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095634 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.095733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095659 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.095733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095673 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.095733 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.095687 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.097016 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.096991 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:07.097262 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.097237 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:24:07.097323 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.097301 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44" (OuterVolumeSpecName: "kube-api-access-tdd44") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "kube-api-access-tdd44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:07.097444 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.097426 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a00bc64-dbf8-497a-8f86-2650ceff85de" (UID: "5a00bc64-dbf8-497a-8f86-2650ceff85de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:07.196245 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.196217 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.196245 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.196242 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.196402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.196252 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdd44\" (UniqueName: \"kubernetes.io/projected/5a00bc64-dbf8-497a-8f86-2650ceff85de-kube-api-access-tdd44\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.196402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.196262 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a00bc64-dbf8-497a-8f86-2650ceff85de-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.196402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.196270 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a00bc64-dbf8-497a-8f86-2650ceff85de-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:07.916840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.916804 2560 generic.go:356] "Generic (PLEG): container finished" podID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerID="4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269" exitCode=0 Sep 16 05:24:07.917051 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.916863 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" Sep 16 05:24:07.917051 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.916873 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerDied","Data":"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269"} Sep 16 05:24:07.917051 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.916923 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247" event={"ID":"5a00bc64-dbf8-497a-8f86-2650ceff85de","Type":"ContainerDied","Data":"f98a1e362bf4b7cf5fcea8d76a5ff893e0782367da67cc09fb1e1481b881aee1"} Sep 16 05:24:07.917051 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.916949 2560 scope.go:117] "RemoveContainer" containerID="4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269" Sep 16 05:24:07.928732 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.928707 2560 scope.go:117] "RemoveContainer" containerID="96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490" Sep 16 05:24:07.935865 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.935841 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:24:07.938151 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.938135 2560 scope.go:117] "RemoveContainer" containerID="4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269" Sep 16 05:24:07.938441 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:07.938422 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269\": container with ID starting with 4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269 not found: ID does not exist" containerID="4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269" Sep 16 05:24:07.938482 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.938452 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269"} err="failed to get container status \"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269\": rpc error: code = NotFound desc = could not find container \"4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269\": container with ID starting with 4f7499fa520f4854077e09c26636cf53a1a5b79291fed25c5b45c13854b58269 not found: ID does not exist" Sep 16 05:24:07.938482 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.938468 2560 scope.go:117] "RemoveContainer" containerID="96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490" Sep 16 05:24:07.938750 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:07.938734 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490\": container with ID starting with 96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490 not found: ID does not exist" containerID="96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490" Sep 16 05:24:07.938809 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.938760 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490"} err="failed to get container status \"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490\": rpc error: code = NotFound desc = could not find container \"96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490\": container with ID starting with 96406b97897809bec7c7fe5212d36eb388e57c19374b62aa869ae97c4050b490 not found: ID does not exist" Sep 16 05:24:07.939875 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:07.939858 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3784e2-openshift-default-68dd7dc58-5x247"] Sep 16 05:24:09.122241 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:09.122209 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" path="/var/lib/kubelet/pods/5a00bc64-dbf8-497a-8f86-2650ceff85de/volumes" Sep 16 05:24:12.609922 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.609889 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:24:12.610433 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.610088 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" podUID="42315721-8b38-4fb1-8bb9-11b51e73748a" containerName="maas-api" containerID="cri-o://a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0" gracePeriod=30 Sep 16 05:24:12.630444 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.630401 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx"] Sep 16 05:24:12.638071 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.638046 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-10ee79-29825595-6g6wx"] Sep 16 05:24:12.858694 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.858670 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:24:12.939247 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.939164 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") pod \"42315721-8b38-4fb1-8bb9-11b51e73748a\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " Sep 16 05:24:12.939247 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.939241 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs\") pod \"42315721-8b38-4fb1-8bb9-11b51e73748a\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " Sep 16 05:24:12.939509 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.939375 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca\") pod \"42315721-8b38-4fb1-8bb9-11b51e73748a\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " Sep 16 05:24:12.939509 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.939424 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5ggr\" (UniqueName: \"kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr\") pod \"42315721-8b38-4fb1-8bb9-11b51e73748a\" (UID: \"42315721-8b38-4fb1-8bb9-11b51e73748a\") " Sep 16 05:24:12.939779 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.939746 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42315721-8b38-4fb1-8bb9-11b51e73748a" (UID: "42315721-8b38-4fb1-8bb9-11b51e73748a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:24:12.941442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.941412 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42315721-8b38-4fb1-8bb9-11b51e73748a" (UID: "42315721-8b38-4fb1-8bb9-11b51e73748a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:24:12.941442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.941432 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42315721-8b38-4fb1-8bb9-11b51e73748a" (UID: "42315721-8b38-4fb1-8bb9-11b51e73748a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:24:12.941624 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.941434 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr" (OuterVolumeSpecName: "kube-api-access-m5ggr") pod "42315721-8b38-4fb1-8bb9-11b51e73748a" (UID: "42315721-8b38-4fb1-8bb9-11b51e73748a"). InnerVolumeSpecName "kube-api-access-m5ggr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:12.948534 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.948507 2560 generic.go:356] "Generic (PLEG): container finished" podID="42315721-8b38-4fb1-8bb9-11b51e73748a" containerID="a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0" exitCode=0 Sep 16 05:24:12.948681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.948577 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" Sep 16 05:24:12.948681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.948594 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" event={"ID":"42315721-8b38-4fb1-8bb9-11b51e73748a","Type":"ContainerDied","Data":"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0"} Sep 16 05:24:12.948681 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.948662 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p" event={"ID":"42315721-8b38-4fb1-8bb9-11b51e73748a","Type":"ContainerDied","Data":"f72b87cb8b6dbcea1d50d381931f6e0b1b9c57c8cf44fd4a5c8bb8ab5999ed34"} Sep 16 05:24:12.948928 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.948695 2560 scope.go:117] "RemoveContainer" containerID="a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0" Sep 16 05:24:12.963095 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.963057 2560 scope.go:117] "RemoveContainer" containerID="a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0" Sep 16 05:24:12.963398 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:12.963341 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0\": container with ID starting with a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0 not found: ID does not exist" containerID="a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0" Sep 16 05:24:12.963398 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.963384 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0"} err="failed to get container status \"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0\": rpc error: code = NotFound desc = could not find container \"a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0\": container with ID starting with a0730be6be06d423b4c76782e45c047049abfed95c9bf0fb08612169bb117ec0 not found: ID does not exist" Sep 16 05:24:12.975534 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.975507 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:24:12.979616 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:12.979590 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-10ee79-594cbd6544-4l26p"] Sep 16 05:24:13.040870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.040831 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:13.040870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.040866 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42315721-8b38-4fb1-8bb9-11b51e73748a-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:13.040870 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.040876 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ggr\" (UniqueName: \"kubernetes.io/projected/42315721-8b38-4fb1-8bb9-11b51e73748a-kube-api-access-m5ggr\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:13.041066 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.040886 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42315721-8b38-4fb1-8bb9-11b51e73748a-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:13.121093 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.121059 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42315721-8b38-4fb1-8bb9-11b51e73748a" path="/var/lib/kubelet/pods/42315721-8b38-4fb1-8bb9-11b51e73748a/volumes" Sep 16 05:24:13.121356 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:13.121344 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe" path="/var/lib/kubelet/pods/bd7a329f-e9ef-4b93-9bca-5bb2aaa778fe/volumes" Sep 16 05:24:19.290038 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:19.289999 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:24:19.290492 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:19.290254 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" containerID="cri-o://74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4" gracePeriod=30 Sep 16 05:24:21.953446 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:21.953401 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 05:24:21.953966 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:21.953698 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59fbdd984-2gfk2" podUID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" containerName="manager" containerID="cri-o://740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f" gracePeriod=30 Sep 16 05:24:22.201235 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.201214 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 05:24:22.321474 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.321392 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert\") pod \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " Sep 16 05:24:22.321665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.321482 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52cw5\" (UniqueName: \"kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5\") pod \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " Sep 16 05:24:22.321665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.321543 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs\") pod \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\" (UID: \"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2\") " Sep 16 05:24:22.323468 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.323438 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5" (OuterVolumeSpecName: "kube-api-access-52cw5") pod "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" (UID: "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2"). InnerVolumeSpecName "kube-api-access-52cw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:22.323596 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.323438 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" (UID: "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:24:22.323596 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.323497 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" (UID: "e8c80e8f-a4f6-4292-89b7-6a2cb16029f2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:24:22.422445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.422410 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:22.422445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.422439 2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-webhook-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:22.422445 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.422448 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52cw5\" (UniqueName: \"kubernetes.io/projected/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2-kube-api-access-52cw5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:22.586947 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.586910 2560 patch_prober.go:28] interesting pod/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 16 05:24:22.587100 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:22.586965 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 05:24:23.007586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.007488 2560 generic.go:356] "Generic (PLEG): container finished" podID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" containerID="740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f" exitCode=0 Sep 16 05:24:23.007586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.007537 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-2gfk2" Sep 16 05:24:23.007586 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.007575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-2gfk2" event={"ID":"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2","Type":"ContainerDied","Data":"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f"} Sep 16 05:24:23.008153 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.007630 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-2gfk2" event={"ID":"e8c80e8f-a4f6-4292-89b7-6a2cb16029f2","Type":"ContainerDied","Data":"30728fb58b996cf07ca41d8c2b89272dcb0787c7ae95b8fe3547232769193460"} Sep 16 05:24:23.008153 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.007654 2560 scope.go:117] "RemoveContainer" containerID="740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f" Sep 16 05:24:23.018286 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.018268 2560 scope.go:117] "RemoveContainer" containerID="740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f" Sep 16 05:24:23.018561 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:23.018530 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f\": container with ID starting with 740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f not found: ID does not exist" containerID="740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f" Sep 16 05:24:23.018620 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.018576 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f"} err="failed to get container status \"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f\": rpc error: code = NotFound desc = could not find container \"740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f\": container with ID starting with 740a808e266cece3de21749dfde07c11c78e2758fe5d658aa07f95352a28645f not found: ID does not exist" Sep 16 05:24:23.029979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.029958 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 05:24:23.035787 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.035761 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fbdd984-2gfk2"] Sep 16 05:24:23.121006 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:23.120981 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" path="/var/lib/kubelet/pods/e8c80e8f-a4f6-4292-89b7-6a2cb16029f2/volumes" Sep 16 05:24:24.541248 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.541223 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:24:24.641057 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641027 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxrmf\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641075 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641103 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641130 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641146 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641216 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641260 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641308 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641337 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data\") pod \"3904c14b-5e48-4338-8f3f-ba52d63f5312\" (UID: \"3904c14b-5e48-4338-8f3f-ba52d63f5312\") " Sep 16 05:24:24.641688 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641439 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:24.641688 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641649 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data" (OuterVolumeSpecName: "istio-data") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:24.641809 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641700 2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.641809 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641798 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:24.641915 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.641834 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:24.642035 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.642017 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:24:24.643376 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.643345 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 05:24:24.643482 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.643389 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 05:24:24.643638 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.643614 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf" (OuterVolumeSpecName: "kube-api-access-hxrmf") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "kube-api-access-hxrmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:24.643638 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.643631 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token" (OuterVolumeSpecName: "istio-token") pod "3904c14b-5e48-4338-8f3f-ba52d63f5312" (UID: "3904c14b-5e48-4338-8f3f-ba52d63f5312"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:24:24.742802 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742767 2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-envoy\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.742802 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742797 2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-token\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.742802 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742806 2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-workload-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.743036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742816 2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-podinfo\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.743036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742824 2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-credential-socket\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.743036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742833 2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3904c14b-5e48-4338-8f3f-ba52d63f5312-istio-data\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.743036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742842 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrmf\" (UniqueName: \"kubernetes.io/projected/3904c14b-5e48-4338-8f3f-ba52d63f5312-kube-api-access-hxrmf\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:24.743036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:24.742850 2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3904c14b-5e48-4338-8f3f-ba52d63f5312-istiod-ca-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:24:25.020684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.020607 2560 generic.go:356] "Generic (PLEG): container finished" podID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerID="74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4" exitCode=0 Sep 16 05:24:25.020684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.020644 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerDied","Data":"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4"} Sep 16 05:24:25.020875 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.020703 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" event={"ID":"3904c14b-5e48-4338-8f3f-ba52d63f5312","Type":"ContainerDied","Data":"dccb9e52d421c0931b92bd74f9deec221b90cd8112877817298419f1bb7a8600"} Sep 16 05:24:25.020875 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.020726 2560 scope.go:117] "RemoveContainer" containerID="74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4" Sep 16 05:24:25.020875 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.020651 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv" Sep 16 05:24:25.030728 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.030710 2560 scope.go:117] "RemoveContainer" containerID="f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac" Sep 16 05:24:25.039146 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.039130 2560 scope.go:117] "RemoveContainer" containerID="74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4" Sep 16 05:24:25.039400 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:25.039380 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4\": container with ID starting with 74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4 not found: ID does not exist" containerID="74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4" Sep 16 05:24:25.039450 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.039410 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4"} err="failed to get container status \"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4\": rpc error: code = NotFound desc = could not find container \"74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4\": container with ID starting with 74f29d376f5b9bc02e79105811268f1fb95fbc937019f7c19c9eaf42d15a02a4 not found: ID does not exist" Sep 16 05:24:25.039450 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.039426 2560 scope.go:117] "RemoveContainer" containerID="f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac" Sep 16 05:24:25.039702 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:24:25.039684 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac\": container with ID starting with f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac not found: ID does not exist" containerID="f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac" Sep 16 05:24:25.039754 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.039706 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac"} err="failed to get container status \"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac\": rpc error: code = NotFound desc = could not find container \"f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac\": container with ID starting with f5afbd317d2807bd8b5b71ae33855ed6737933ed59bb166fd2c673af82d715ac not found: ID does not exist" Sep 16 05:24:25.044245 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.044219 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:24:25.046638 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.046621 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-10ee79-openshift-default-574c98d696-kznjv"] Sep 16 05:24:25.121100 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:25.121070 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" path="/var/lib/kubelet/pods/3904c14b-5e48-4338-8f3f-ba52d63f5312/volumes" Sep 16 05:24:55.394918 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:24:55.394889 2560 scope.go:117] "RemoveContainer" containerID="67cb618db79a724ae435e77ea959c324314ed8bbf032bf285d58b74dccc7b7a1" Sep 16 05:25:23.821245 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821205 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:23.821793 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821768 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" containerName="manager" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821796 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" containerName="manager" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821810 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821819 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821837 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bb45fa-4854-4d27-a922-107003a1e30b" containerName="maas-api" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821846 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bb45fa-4854-4d27-a922-107003a1e30b" containerName="maas-api" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821855 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821864 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:23.821874 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821873 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42315721-8b38-4fb1-8bb9-11b51e73748a" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821882 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42315721-8b38-4fb1-8bb9-11b51e73748a" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821911 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821920 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821945 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.821953 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822046 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822059 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822069 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bb45fa-4854-4d27-a922-107003a1e30b" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822081 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c80e8f-a4f6-4292-89b7-6a2cb16029f2" containerName="manager" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822089 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42315721-8b38-4fb1-8bb9-11b51e73748a" containerName="maas-api" Sep 16 05:25:23.822272 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.822098 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7449ecf0-7357-47ff-b7a5-80a2ae2ecb8b" containerName="maas-api" Sep 16 05:25:23.825430 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.825407 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:23.828111 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.828083 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 05:25:23.829266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.829244 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 05:25:23.829266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.829261 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mx82w\"" Sep 16 05:25:23.829416 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.829290 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 05:25:23.835218 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.835193 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:23.996914 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.996881 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:23.997088 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.996945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqcf\" (UniqueName: \"kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:23.997088 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:23.996984 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.097840 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.097808 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.098003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.097864 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.098003 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.097921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqcf\" (UniqueName: \"kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.100475 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.100441 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.100605 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.100441 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.105777 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.105756 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqcf\" (UniqueName: \"kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf\") pod \"maas-controller-59fbdd984-qjfh9\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.137865 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.137826 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:24.412439 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:24.412409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:24.414114 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:25:24.414088 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ec3019_1f35_4e75_bc79_c4928ef80218.slice/crio-3d7b3939a4e2109c90e463b450e55be20b3497f60faa0b468f93519ca42d0565 WatchSource:0}: Error finding container 3d7b3939a4e2109c90e463b450e55be20b3497f60faa0b468f93519ca42d0565: Status 404 returned error can't find the container with id 3d7b3939a4e2109c90e463b450e55be20b3497f60faa0b468f93519ca42d0565 Sep 16 05:25:25.358665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:25.358596 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-qjfh9" event={"ID":"42ec3019-1f35-4e75-bc79-c4928ef80218","Type":"ContainerStarted","Data":"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba"} Sep 16 05:25:25.359086 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:25.358689 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-qjfh9" event={"ID":"42ec3019-1f35-4e75-bc79-c4928ef80218","Type":"ContainerStarted","Data":"3d7b3939a4e2109c90e463b450e55be20b3497f60faa0b468f93519ca42d0565"} Sep 16 05:25:25.359086 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:25.358815 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:25.376999 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:25.376952 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59fbdd984-qjfh9" podStartSLOduration=1.948405629 podStartE2EDuration="2.37693598s" totalImagesPullingTime="428.530351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:25:25.375817599 +0000 UTC m=+2190.889369134" watchObservedRunningTime="2026-09-16 05:25:25.37693598 +0000 UTC m=+2190.890487514" Sep 16 05:25:36.368041 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:36.368010 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:43.537085 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.537053 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:43.537535 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.537286 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59fbdd984-qjfh9" podUID="42ec3019-1f35-4e75-bc79-c4928ef80218" containerName="manager" containerID="cri-o://268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba" gracePeriod=30 Sep 16 05:25:43.780065 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.780040 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:43.868393 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.868363 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert\") pod \"42ec3019-1f35-4e75-bc79-c4928ef80218\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " Sep 16 05:25:43.868563 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.868465 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs\") pod \"42ec3019-1f35-4e75-bc79-c4928ef80218\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " Sep 16 05:25:43.868563 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.868529 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mqcf\" (UniqueName: \"kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf\") pod \"42ec3019-1f35-4e75-bc79-c4928ef80218\" (UID: \"42ec3019-1f35-4e75-bc79-c4928ef80218\") " Sep 16 05:25:43.870439 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.870411 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "42ec3019-1f35-4e75-bc79-c4928ef80218" (UID: "42ec3019-1f35-4e75-bc79-c4928ef80218"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:25:43.870583 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.870463 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42ec3019-1f35-4e75-bc79-c4928ef80218" (UID: "42ec3019-1f35-4e75-bc79-c4928ef80218"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:25:43.870583 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.870497 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf" (OuterVolumeSpecName: "kube-api-access-8mqcf") pod "42ec3019-1f35-4e75-bc79-c4928ef80218" (UID: "42ec3019-1f35-4e75-bc79-c4928ef80218"). InnerVolumeSpecName "kube-api-access-8mqcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:25:43.969766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.969731 2560 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-webhook-cert\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:25:43.969766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.969761 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42ec3019-1f35-4e75-bc79-c4928ef80218-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:25:43.969766 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:43.969771 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqcf\" (UniqueName: \"kubernetes.io/projected/42ec3019-1f35-4e75-bc79-c4928ef80218-kube-api-access-8mqcf\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:25:44.465667 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.465636 2560 generic.go:356] "Generic (PLEG): container finished" podID="42ec3019-1f35-4e75-bc79-c4928ef80218" containerID="268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba" exitCode=0 Sep 16 05:25:44.465836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.465692 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-qjfh9" event={"ID":"42ec3019-1f35-4e75-bc79-c4928ef80218","Type":"ContainerDied","Data":"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba"} Sep 16 05:25:44.465836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.465695 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-qjfh9" Sep 16 05:25:44.465836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.465734 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-qjfh9" event={"ID":"42ec3019-1f35-4e75-bc79-c4928ef80218","Type":"ContainerDied","Data":"3d7b3939a4e2109c90e463b450e55be20b3497f60faa0b468f93519ca42d0565"} Sep 16 05:25:44.465836 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.465757 2560 scope.go:117] "RemoveContainer" containerID="268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba" Sep 16 05:25:44.475365 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.475339 2560 scope.go:117] "RemoveContainer" containerID="268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba" Sep 16 05:25:44.475651 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:25:44.475632 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba\": container with ID starting with 268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba not found: ID does not exist" containerID="268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba" Sep 16 05:25:44.475731 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.475657 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba"} err="failed to get container status \"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba\": rpc error: code = NotFound desc = could not find container \"268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba\": container with ID starting with 268bbc031a8eeef9e0d6cd230f73a425c897933ac0f8bea00f26e10c313088ba not found: ID does not exist" Sep 16 05:25:44.488330 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.488307 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:44.492031 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:44.492010 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59fbdd984-qjfh9"] Sep 16 05:25:45.123153 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.123121 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ec3019-1f35-4e75-bc79-c4928ef80218" path="/var/lib/kubelet/pods/42ec3019-1f35-4e75-bc79-c4928ef80218/volumes" Sep 16 05:25:45.224776 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.224742 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59fbdd984-ctkdn"] Sep 16 05:25:45.225107 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225095 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ec3019-1f35-4e75-bc79-c4928ef80218" containerName="manager" Sep 16 05:25:45.225107 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225108 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ec3019-1f35-4e75-bc79-c4928ef80218" containerName="manager" Sep 16 05:25:45.225189 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225127 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:45.225189 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225133 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:45.225264 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225193 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="42ec3019-1f35-4e75-bc79-c4928ef80218" containerName="manager" Sep 16 05:25:45.225264 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225203 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3904c14b-5e48-4338-8f3f-ba52d63f5312" containerName="istio-proxy" Sep 16 05:25:45.225264 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.225210 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a00bc64-dbf8-497a-8f86-2650ceff85de" containerName="istio-proxy" Sep 16 05:25:45.229534 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.229518 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.233326 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.233301 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 16 05:25:45.233455 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.233308 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 16 05:25:45.233598 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.233582 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 16 05:25:45.233665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.233649 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mx82w\"" Sep 16 05:25:45.238333 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.238310 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-ctkdn"] Sep 16 05:25:45.281212 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.281185 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6qb\" (UniqueName: \"kubernetes.io/projected/4718d342-54ae-4ac9-8252-efb1ae776527-kube-api-access-9s6qb\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.281343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.281238 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-webhook-cert\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.281343 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.281294 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-metrics-certs\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.381915 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.381837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-webhook-cert\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.381915 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.381882 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-metrics-certs\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.382079 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.381939 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6qb\" (UniqueName: \"kubernetes.io/projected/4718d342-54ae-4ac9-8252-efb1ae776527-kube-api-access-9s6qb\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.384314 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.384286 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-webhook-cert\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.384442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.384387 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4718d342-54ae-4ac9-8252-efb1ae776527-metrics-certs\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.390249 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.390227 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6qb\" (UniqueName: \"kubernetes.io/projected/4718d342-54ae-4ac9-8252-efb1ae776527-kube-api-access-9s6qb\") pod \"maas-controller-59fbdd984-ctkdn\" (UID: \"4718d342-54ae-4ac9-8252-efb1ae776527\") " pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.540767 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.540737 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:45.816032 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.816008 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59fbdd984-ctkdn"] Sep 16 05:25:45.817692 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:25:45.817657 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4718d342_54ae_4ac9_8252_efb1ae776527.slice/crio-7de15c4f2781410a63edd6c3f252feae51920a807cfe8700a03d27e44a21e68a WatchSource:0}: Error finding container 7de15c4f2781410a63edd6c3f252feae51920a807cfe8700a03d27e44a21e68a: Status 404 returned error can't find the container with id 7de15c4f2781410a63edd6c3f252feae51920a807cfe8700a03d27e44a21e68a Sep 16 05:25:45.818882 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:45.818864 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 05:25:46.480440 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:46.480399 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-ctkdn" event={"ID":"4718d342-54ae-4ac9-8252-efb1ae776527","Type":"ContainerStarted","Data":"6831b7b391e5c5ac7bb40ef3584a24256d977ce68370b65c38eb54642b98bea9"} Sep 16 05:25:46.480811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:46.480464 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:25:46.480811 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:46.480481 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59fbdd984-ctkdn" event={"ID":"4718d342-54ae-4ac9-8252-efb1ae776527","Type":"ContainerStarted","Data":"7de15c4f2781410a63edd6c3f252feae51920a807cfe8700a03d27e44a21e68a"} Sep 16 05:25:46.500769 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:46.500722 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59fbdd984-ctkdn" podStartSLOduration=1.031704973 podStartE2EDuration="1.500705861s" totalImagesPullingTime="469.000888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:25:46.499074209 +0000 UTC m=+2212.012625742" watchObservedRunningTime="2026-09-16 05:25:46.500705861 +0000 UTC m=+2212.014257418" Sep 16 05:25:57.489176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:25:57.489146 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59fbdd984-ctkdn" Sep 16 05:28:55.256665 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:28:55.256638 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:28:55.259580 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:28:55.259542 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:28:55.262715 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:28:55.262693 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:28:55.265577 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:28:55.265540 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:29:52.353166 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.353131 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f"] Sep 16 05:29:52.356831 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.356809 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.367848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.367824 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f"] Sep 16 05:29:52.509785 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.509754 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7mrv\" (UniqueName: \"kubernetes.io/projected/76063cc3-64b4-4928-93ce-5fa7fc7501c7-kube-api-access-z7mrv\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.509785 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.509789 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76063cc3-64b4-4928-93ce-5fa7fc7501c7-openshift-service-ca\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.509977 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.509813 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-maas-api-tls\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.509977 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.509917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-metrics-certs\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.610878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.610802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7mrv\" (UniqueName: \"kubernetes.io/projected/76063cc3-64b4-4928-93ce-5fa7fc7501c7-kube-api-access-z7mrv\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.610878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.610837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76063cc3-64b4-4928-93ce-5fa7fc7501c7-openshift-service-ca\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.610878 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.610861 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-maas-api-tls\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.611083 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.610898 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-metrics-certs\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.611567 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.611531 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76063cc3-64b4-4928-93ce-5fa7fc7501c7-openshift-service-ca\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.613241 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.613221 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-metrics-certs\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.613339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.613282 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76063cc3-64b4-4928-93ce-5fa7fc7501c7-maas-api-tls\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.619161 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.619143 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7mrv\" (UniqueName: \"kubernetes.io/projected/76063cc3-64b4-4928-93ce-5fa7fc7501c7-kube-api-access-z7mrv\") pod \"maas-api-5c4458d669-c2f6f\" (UID: \"76063cc3-64b4-4928-93ce-5fa7fc7501c7\") " pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.668024 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.668000 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:52.946675 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:52.946643 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f"] Sep 16 05:29:52.948051 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:29:52.948027 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76063cc3_64b4_4928_93ce_5fa7fc7501c7.slice/crio-1cda126a2f0987b9f007e5e152bdaa0dd26f736e4c48a4d112ec84c3a5590aed WatchSource:0}: Error finding container 1cda126a2f0987b9f007e5e152bdaa0dd26f736e4c48a4d112ec84c3a5590aed: Status 404 returned error can't find the container with id 1cda126a2f0987b9f007e5e152bdaa0dd26f736e4c48a4d112ec84c3a5590aed Sep 16 05:29:53.855489 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:53.855449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" event={"ID":"76063cc3-64b4-4928-93ce-5fa7fc7501c7","Type":"ContainerStarted","Data":"1616acc53657e10f83605409a197d00ea3f8a653738f627ad804a34a526ed238"} Sep 16 05:29:53.855991 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:53.855502 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" event={"ID":"76063cc3-64b4-4928-93ce-5fa7fc7501c7","Type":"ContainerStarted","Data":"1cda126a2f0987b9f007e5e152bdaa0dd26f736e4c48a4d112ec84c3a5590aed"} Sep 16 05:29:53.855991 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:53.855530 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:53.875366 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:53.875314 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" podStartSLOduration=1.546028043 podStartE2EDuration="1.875299074s" totalImagesPullingTime="329.271031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:29:53.871795447 +0000 UTC m=+2459.385346998" watchObservedRunningTime="2026-09-16 05:29:53.875299074 +0000 UTC m=+2459.388850608" Sep 16 05:29:59.867463 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:59.867432 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c4458d669-c2f6f" Sep 16 05:29:59.922579 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:59.922491 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:29:59.922854 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:29:59.922797 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" podUID="4454818e-aca4-43c6-ba71-0657a50b749b" containerName="maas-api" containerID="cri-o://d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a" gracePeriod=30 Sep 16 05:30:01.171156 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.171132 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:30:01.181121 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.181094 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca\") pod \"4454818e-aca4-43c6-ba71-0657a50b749b\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " Sep 16 05:30:01.181233 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.181155 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs\") pod \"4454818e-aca4-43c6-ba71-0657a50b749b\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " Sep 16 05:30:01.181233 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.181181 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") pod \"4454818e-aca4-43c6-ba71-0657a50b749b\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " Sep 16 05:30:01.181337 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.181249 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sljf5\" (UniqueName: \"kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5\") pod \"4454818e-aca4-43c6-ba71-0657a50b749b\" (UID: \"4454818e-aca4-43c6-ba71-0657a50b749b\") " Sep 16 05:30:01.181489 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.181463 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4454818e-aca4-43c6-ba71-0657a50b749b" (UID: "4454818e-aca4-43c6-ba71-0657a50b749b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 05:30:01.183155 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.183132 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4454818e-aca4-43c6-ba71-0657a50b749b" (UID: "4454818e-aca4-43c6-ba71-0657a50b749b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:30:01.183255 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.183226 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5" (OuterVolumeSpecName: "kube-api-access-sljf5") pod "4454818e-aca4-43c6-ba71-0657a50b749b" (UID: "4454818e-aca4-43c6-ba71-0657a50b749b"). InnerVolumeSpecName "kube-api-access-sljf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 05:30:01.183325 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.183302 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4454818e-aca4-43c6-ba71-0657a50b749b" (UID: "4454818e-aca4-43c6-ba71-0657a50b749b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 05:30:01.282050 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.281972 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4454818e-aca4-43c6-ba71-0657a50b749b-openshift-service-ca\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:30:01.282050 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.282000 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-metrics-certs\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:30:01.282050 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.282009 2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4454818e-aca4-43c6-ba71-0657a50b749b-maas-api-tls\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:30:01.282050 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.282018 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sljf5\" (UniqueName: \"kubernetes.io/projected/4454818e-aca4-43c6-ba71-0657a50b749b-kube-api-access-sljf5\") on node \"ip-10-0-142-144.ec2.internal\" DevicePath \"\"" Sep 16 05:30:01.899790 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.899757 2560 generic.go:356] "Generic (PLEG): container finished" podID="4454818e-aca4-43c6-ba71-0657a50b749b" containerID="d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a" exitCode=0 Sep 16 05:30:01.899955 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.899802 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" Sep 16 05:30:01.899955 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.899837 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" event={"ID":"4454818e-aca4-43c6-ba71-0657a50b749b","Type":"ContainerDied","Data":"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a"} Sep 16 05:30:01.899955 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.899890 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662" event={"ID":"4454818e-aca4-43c6-ba71-0657a50b749b","Type":"ContainerDied","Data":"f62ced7d42c277c7f231523d187afd3c1573bed55f5694cc0fba0cbe9996e918"} Sep 16 05:30:01.899955 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.899916 2560 scope.go:117] "RemoveContainer" containerID="d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a" Sep 16 05:30:01.911769 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.911748 2560 scope.go:117] "RemoveContainer" containerID="d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a" Sep 16 05:30:01.912018 ip-10-0-142-144 kubenswrapper[2560]: E0916 05:30:01.912003 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a\": container with ID starting with d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a not found: ID does not exist" containerID="d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a" Sep 16 05:30:01.912073 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.912025 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a"} err="failed to get container status \"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a\": rpc error: code = NotFound desc = could not find container \"d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a\": container with ID starting with d9c16d1b9d40c8b3df1e28db081268e7cbdd545bfc3c5d2e6929fc0c743e7e8a not found: ID does not exist" Sep 16 05:30:01.926416 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.926394 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:30:01.928862 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:01.928839 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4f95b8f-vq662"] Sep 16 05:30:03.121593 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:03.121535 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4454818e-aca4-43c6-ba71-0657a50b749b" path="/var/lib/kubelet/pods/4454818e-aca4-43c6-ba71-0657a50b749b/volumes" Sep 16 05:30:14.272848 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:14.272813 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d444b47b-957qw_b3e80c37-f0da-44b9-af00-f4fe2265e8d4/authorino/0.log" Sep 16 05:30:19.655769 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:19.655735 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fbdd984-ctkdn_4718d342-54ae-4ac9-8252-efb1ae776527/manager/0.log" Sep 16 05:30:20.260269 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:20.260236 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-zbnrd_0cbf593d-cef9-4a7f-8dd1-86fdf372811a/copy-manifests/0.log" Sep 16 05:30:20.266267 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:20.266243 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-zbnrd_0cbf593d-cef9-4a7f-8dd1-86fdf372811a/manager/0.log" Sep 16 05:30:20.612684 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:20.612653 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c4458d669-c2f6f_76063cc3-64b4-4928-93ce-5fa7fc7501c7/maas-api/0.log" Sep 16 05:30:21.842191 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:21.842139 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d444b47b-957qw_b3e80c37-f0da-44b9-af00-f4fe2265e8d4/authorino/0.log" Sep 16 05:30:22.183814 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:22.183738 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-blzpq_0789c1c8-920e-4ef2-acbf-7168e7cf3b26/kuadrant-console-plugin/0.log" Sep 16 05:30:23.129054 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:23.129024 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-6b9hl_495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde/kube-auth-proxy/0.log" Sep 16 05:30:23.716451 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:23.716400 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75c67795c4-rktps_94700653-88d4-48d5-9d5a-4ba8174af04f/router/0.log" Sep 16 05:30:24.063025 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:24.062937 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-29nwc_abcf1429-ec8b-4377-aa14-438236a2df38/main/0.log" Sep 16 05:30:24.379409 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:24.379369 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qq9jx_ade66949-a4fe-4ffe-801a-c5c56ba3303e/main/0.log" Sep 16 05:30:29.793294 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.793263 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hf662/must-gather-hr4w9"] Sep 16 05:30:29.793867 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.793741 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4454818e-aca4-43c6-ba71-0657a50b749b" containerName="maas-api" Sep 16 05:30:29.793867 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.793757 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4454818e-aca4-43c6-ba71-0657a50b749b" containerName="maas-api" Sep 16 05:30:29.793867 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.793836 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4454818e-aca4-43c6-ba71-0657a50b749b" containerName="maas-api" Sep 16 05:30:29.797394 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.797374 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:29.800159 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.800133 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hf662\"/\"default-dockercfg-p95kz\"" Sep 16 05:30:29.800266 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.800206 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hf662\"/\"openshift-service-ca.crt\"" Sep 16 05:30:29.801286 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.801273 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hf662\"/\"kube-root-ca.crt\"" Sep 16 05:30:29.807898 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.807877 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/must-gather-hr4w9"] Sep 16 05:30:29.937089 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.937055 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8de9a43-4b3c-4a93-b30b-efb110326ffa-must-gather-output\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:29.937247 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:29.937122 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7jh\" (UniqueName: \"kubernetes.io/projected/e8de9a43-4b3c-4a93-b30b-efb110326ffa-kube-api-access-zh7jh\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.038205 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.038177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8de9a43-4b3c-4a93-b30b-efb110326ffa-must-gather-output\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.038398 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.038235 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7jh\" (UniqueName: \"kubernetes.io/projected/e8de9a43-4b3c-4a93-b30b-efb110326ffa-kube-api-access-zh7jh\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.038594 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.038543 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e8de9a43-4b3c-4a93-b30b-efb110326ffa-must-gather-output\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.046444 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.046388 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7jh\" (UniqueName: \"kubernetes.io/projected/e8de9a43-4b3c-4a93-b30b-efb110326ffa-kube-api-access-zh7jh\") pod \"must-gather-hr4w9\" (UID: \"e8de9a43-4b3c-4a93-b30b-efb110326ffa\") " pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.107402 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.107378 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/must-gather-hr4w9" Sep 16 05:30:30.378847 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:30.378823 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/must-gather-hr4w9"] Sep 16 05:30:30.380048 ip-10-0-142-144 kubenswrapper[2560]: W0916 05:30:30.380008 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8de9a43_4b3c_4a93_b30b_efb110326ffa.slice/crio-a32646112a58d5114f90d64658ef4d48ec986b9431e0002d58ef427ed3d9985c WatchSource:0}: Error finding container a32646112a58d5114f90d64658ef4d48ec986b9431e0002d58ef427ed3d9985c: Status 404 returned error can't find the container with id a32646112a58d5114f90d64658ef4d48ec986b9431e0002d58ef427ed3d9985c Sep 16 05:30:31.063154 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:31.063117 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/must-gather-hr4w9" event={"ID":"e8de9a43-4b3c-4a93-b30b-efb110326ffa","Type":"ContainerStarted","Data":"a32646112a58d5114f90d64658ef4d48ec986b9431e0002d58ef427ed3d9985c"} Sep 16 05:30:32.071442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:32.071388 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/must-gather-hr4w9" event={"ID":"e8de9a43-4b3c-4a93-b30b-efb110326ffa","Type":"ContainerStarted","Data":"6f074404fc9e9c86ce538df37e651c6690f9d5e2f80f207149e47289b2322882"} Sep 16 05:30:32.071442 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:32.071449 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/must-gather-hr4w9" event={"ID":"e8de9a43-4b3c-4a93-b30b-efb110326ffa","Type":"ContainerStarted","Data":"292c067abb83825bdd4c31bc599a4c7917c6349feffc2a36a14493ec656dca80"} Sep 16 05:30:32.090567 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:32.090494 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hf662/must-gather-hr4w9" podStartSLOduration=2.177384136 podStartE2EDuration="3.090478022s" totalImagesPullingTime="913.093886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 05:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:30:32.08737922 +0000 UTC m=+2497.600930766" watchObservedRunningTime="2026-09-16 05:30:32.090478022 +0000 UTC m=+2497.604029555" Sep 16 05:30:32.772428 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:32.772399 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4jhmn_20993e50-9419-404f-a4b6-0eda4a2afeff/global-pull-secret-syncer/0.log" Sep 16 05:30:32.920683 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:32.920644 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9d9xk_b5a7b348-8687-44f7-9e58-411bb466c01e/konnectivity-agent/0.log" Sep 16 05:30:33.024093 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:33.024000 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-144.ec2.internal_3871b89cd4ee6045bc4279bdebdf12fc/haproxy/0.log" Sep 16 05:30:37.571334 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:37.570978 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-86d444b47b-957qw_b3e80c37-f0da-44b9-af00-f4fe2265e8d4/authorino/0.log" Sep 16 05:30:37.658406 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:37.658379 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-blzpq_0789c1c8-920e-4ef2-acbf-7168e7cf3b26/kuadrant-console-plugin/0.log" Sep 16 05:30:39.425516 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.425397 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-sfq6k_1e08416d-00b8-4621-bd10-c653c8e424e3/cluster-monitoring-operator/0.log" Sep 16 05:30:39.450280 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.450249 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q62kf_8b61f1cf-bad9-45db-b9ac-5c4a67851a75/kube-state-metrics/0.log" Sep 16 05:30:39.466823 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.466744 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q62kf_8b61f1cf-bad9-45db-b9ac-5c4a67851a75/kube-rbac-proxy-main/0.log" Sep 16 05:30:39.485603 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.485571 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-q62kf_8b61f1cf-bad9-45db-b9ac-5c4a67851a75/kube-rbac-proxy-self/0.log" Sep 16 05:30:39.511174 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.511149 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d46fd8d8f-pgbrp_3d43a618-7716-4aec-8efa-30dbe024f733/metrics-server/0.log" Sep 16 05:30:39.543183 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.543157 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-fs7hj_517a0a44-a36c-42cf-bc5e-12b71a8838c7/monitoring-plugin/0.log" Sep 16 05:30:39.710639 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.710478 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqf6r_e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a/node-exporter/0.log" Sep 16 05:30:39.729260 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.729219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqf6r_e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a/kube-rbac-proxy/0.log" Sep 16 05:30:39.747866 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.747837 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tqf6r_e99aa397-ad3f-4fef-a8ad-c633f1b7ab4a/init-textfile/0.log" Sep 16 05:30:39.775381 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.775301 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pkbdz_a0979b33-ec33-4fb0-8e9f-043991d0e998/kube-rbac-proxy-main/0.log" Sep 16 05:30:39.795423 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.795380 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pkbdz_a0979b33-ec33-4fb0-8e9f-043991d0e998/kube-rbac-proxy-self/0.log" Sep 16 05:30:39.819601 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:39.819573 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pkbdz_a0979b33-ec33-4fb0-8e9f-043991d0e998/openshift-state-metrics/0.log" Sep 16 05:30:40.095042 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:40.095011 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74fdb4b7fc-8sthr_92d1a55b-289a-49a8-b612-3f9e4c1c806d/telemeter-client/0.log" Sep 16 05:30:40.114866 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:40.114839 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74fdb4b7fc-8sthr_92d1a55b-289a-49a8-b612-3f9e4c1c806d/reload/0.log" Sep 16 05:30:40.142392 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:40.142358 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-74fdb4b7fc-8sthr_92d1a55b-289a-49a8-b612-3f9e4c1c806d/kube-rbac-proxy/0.log" Sep 16 05:30:41.810859 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:41.810826 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-bg5v5_da526990-e05b-4d59-b01a-217871f25d23/networking-console-plugin/0.log" Sep 16 05:30:42.437118 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:42.437082 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/1.log" Sep 16 05:30:42.443059 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:42.443004 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-vpvwx_d8f5f528-9fe1-4dcd-a916-f2db73ae6d96/console-operator/2.log" Sep 16 05:30:42.979435 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:42.979403 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54956d744f-2jn8f_7f259120-d360-468d-87a6-4ab7e01864a5/console/0.log" Sep 16 05:30:43.409138 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.409100 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts"] Sep 16 05:30:43.418753 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.415191 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.423423 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.423389 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts"] Sep 16 05:30:43.498773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.498722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-lib-modules\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.498970 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.498819 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-proc\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.498970 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.498862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-sys\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.499096 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.499006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnkj\" (UniqueName: \"kubernetes.io/projected/d1864778-00ea-47e1-a5a5-4480e31ed3ef-kube-api-access-sjnkj\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.499096 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.499052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-podres\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600436 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600387 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-sys\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600521 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnkj\" (UniqueName: \"kubernetes.io/projected/d1864778-00ea-47e1-a5a5-4480e31ed3ef-kube-api-access-sjnkj\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600525 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-sys\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600655 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600582 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-podres\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600838 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-lib-modules\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600838 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600796 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-podres\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.600838 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600805 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-lib-modules\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.601009 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600879 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-proc\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.601009 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.600957 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1864778-00ea-47e1-a5a5-4480e31ed3ef-proc\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.609340 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.609296 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnkj\" (UniqueName: \"kubernetes.io/projected/d1864778-00ea-47e1-a5a5-4480e31ed3ef-kube-api-access-sjnkj\") pod \"perf-node-gather-daemonset-h2sts\" (UID: \"d1864778-00ea-47e1-a5a5-4480e31ed3ef\") " pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:43.699090 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.699006 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-krqql_c3953222-bec7-4f4c-80c3-2b51fd9b7ebe/volume-data-source-validator/0.log" Sep 16 05:30:43.740732 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:43.740696 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:44.053888 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:44.051409 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts"] Sep 16 05:30:44.177542 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:44.177510 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" event={"ID":"d1864778-00ea-47e1-a5a5-4480e31ed3ef","Type":"ContainerStarted","Data":"0208a7708569568d44ef563732e17d3cde9c2a3704b31fc5479fa80401654516"} Sep 16 05:30:44.990281 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:44.990248 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jswl4_195e625f-1f88-40b1-848b-8efbaff69d33/dns/0.log" Sep 16 05:30:45.006110 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.006077 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jswl4_195e625f-1f88-40b1-848b-8efbaff69d33/kube-rbac-proxy/0.log" Sep 16 05:30:45.066956 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.066922 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fx8ph_695af7a0-6dde-4d7d-b185-32ef2b342741/dns-node-resolver/0.log" Sep 16 05:30:45.186635 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.186603 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" event={"ID":"d1864778-00ea-47e1-a5a5-4480e31ed3ef","Type":"ContainerStarted","Data":"1e5884692cb4a650ba7fe43437c4a8c6b81c1df93dd22b8db872a948a45d7c15"} Sep 16 05:30:45.186793 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.186739 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:45.204997 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.204948 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" podStartSLOduration=2.2049341399999998 podStartE2EDuration="2.20493414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 05:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 05:30:45.202639565 +0000 UTC m=+2510.716191106" watchObservedRunningTime="2026-09-16 05:30:45.20493414 +0000 UTC m=+2510.718485721" Sep 16 05:30:45.727623 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:45.727593 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vl57t_6155ff74-4c91-44f2-8bd7-6768c8fe3de8/node-ca/0.log" Sep 16 05:30:47.034726 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:47.034700 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-6b9hl_495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde/kube-auth-proxy/0.log" Sep 16 05:30:47.206773 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:47.206740 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75c67795c4-rktps_94700653-88d4-48d5-9d5a-4ba8174af04f/router/0.log" Sep 16 05:30:48.105144 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.105072 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qkwgl_f13e2f9d-cfab-478b-9ad1-58f6dd8c0336/serve-healthcheck-canary/0.log" Sep 16 05:30:48.519231 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.519156 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9l49f_31d4c6fc-9a85-49bd-b99d-bd839996e36d/insights-operator/0.log" Sep 16 05:30:48.521269 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.521242 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9l49f_31d4c6fc-9a85-49bd-b99d-bd839996e36d/insights-operator/1.log" Sep 16 05:30:48.601176 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.601152 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7fmq_6a86b362-5097-4439-8d31-69ef7ccce013/kube-rbac-proxy/0.log" Sep 16 05:30:48.616979 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.616951 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7fmq_6a86b362-5097-4439-8d31-69ef7ccce013/exporter/0.log" Sep 16 05:30:48.634257 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:48.634230 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p7fmq_6a86b362-5097-4439-8d31-69ef7ccce013/extractor/0.log" Sep 16 05:30:51.204980 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:51.204943 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hf662/perf-node-gather-daemonset-h2sts" Sep 16 05:30:51.287707 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:51.287673 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59fbdd984-ctkdn_4718d342-54ae-4ac9-8252-efb1ae776527/manager/0.log" Sep 16 05:30:51.580388 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:51.580357 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-zbnrd_0cbf593d-cef9-4a7f-8dd1-86fdf372811a/manager/0.log" Sep 16 05:30:51.597053 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:51.597022 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-756589c867-zbnrd_0cbf593d-cef9-4a7f-8dd1-86fdf372811a/copy-manifests/0.log" Sep 16 05:30:52.955452 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:52.955426 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-778cf7776f-xtqcq_32e03a38-2b05-4bda-abf4-1d67b749d622/manager/0.log" Sep 16 05:30:53.611220 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:53.611196 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-6b9hl_495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde/kube-auth-proxy/0.log" Sep 16 05:30:53.723100 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:53.723069 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75c67795c4-rktps_94700653-88d4-48d5-9d5a-4ba8174af04f/router/0.log" Sep 16 05:30:55.708503 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:55.708477 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6965d5747b-6b9hl_495fa8f0-2ec0-4dd3-837d-8b3f1da8ccde/kube-auth-proxy/0.log" Sep 16 05:30:55.832534 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:55.832503 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75c67795c4-rktps_94700653-88d4-48d5-9d5a-4ba8174af04f/router/0.log" Sep 16 05:30:58.287485 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:58.287453 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-qf8dt_b6074ca0-4d90-47b2-8c5e-069b750b9fd9/kube-storage-version-migrator-operator/1.log" Sep 16 05:30:58.288256 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:58.288232 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-qf8dt_b6074ca0-4d90-47b2-8c5e-069b750b9fd9/kube-storage-version-migrator-operator/0.log" Sep 16 05:30:59.530990 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.530963 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/kube-multus-additional-cni-plugins/0.log" Sep 16 05:30:59.548339 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.548314 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/egress-router-binary-copy/0.log" Sep 16 05:30:59.564843 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.564821 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/cni-plugins/0.log" Sep 16 05:30:59.583101 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.583083 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/bond-cni-plugin/0.log" Sep 16 05:30:59.599036 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.599013 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/routeoverride-cni/0.log" Sep 16 05:30:59.615264 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.615242 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/whereabouts-cni-bincopy/0.log" Sep 16 05:30:59.632962 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.632941 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d4rtn_7439855c-a586-4e18-b668-31e119086964/whereabouts-cni/0.log" Sep 16 05:30:59.814268 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.814190 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cz7c2_8523d131-96e7-48c9-bc8c-4a99a787d299/kube-multus/0.log" Sep 16 05:30:59.833147 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.833120 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gknpb_70065c80-52d4-4eb5-a606-905ee57917a9/network-metrics-daemon/0.log" Sep 16 05:30:59.847716 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:30:59.847693 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gknpb_70065c80-52d4-4eb5-a606-905ee57917a9/kube-rbac-proxy/0.log" Sep 16 05:31:00.760784 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.760747 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-controller/0.log" Sep 16 05:31:00.776458 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.776417 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/0.log" Sep 16 05:31:00.776626 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.776499 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/ovn-acl-logging/1.log" Sep 16 05:31:00.795131 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.795102 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/kube-rbac-proxy-node/0.log" Sep 16 05:31:00.815652 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.815630 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 05:31:00.831441 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.831414 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/northd/0.log" Sep 16 05:31:00.853892 ip-10-0-142-144 kubenswrapper[2560]: I0916 05:31:00.853860 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-r2bcd_6ed9bf57-c12e-499c-b4e3-abfd955aa783/nbdb/0.log"