Sep 14 01:30:53.225545 ip-10-0-132-205 systemd[1]: Starting Kubernetes Kubelet... Sep 14 01:30:53.648389 ip-10-0-132-205 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:53.648389 ip-10-0-132-205 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 01:30:53.648389 ip-10-0-132-205 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:53.648389 ip-10-0-132-205 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 01:30:53.648389 ip-10-0-132-205 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 01:30:53.649471 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.649384 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 01:30:53.654869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654853 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.654869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654869 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654874 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654878 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654881 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654884 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654887 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654890 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654893 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654896 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654899 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654901 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654904 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654907 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654909 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654912 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654915 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654917 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654920 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654923 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.654936 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654931 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654933 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654936 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654939 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654942 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654944 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654947 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654950 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654952 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654955 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654958 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654961 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654963 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654966 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654969 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654971 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654974 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654976 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654979 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.655424 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654981 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654984 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654987 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654989 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654993 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654995 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.654998 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655001 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655003 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655006 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655008 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655010 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655013 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655015 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655018 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655020 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655024 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655027 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655030 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655032 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.655880 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655035 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655038 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655041 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655043 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655046 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655048 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655051 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655053 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655056 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655058 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655061 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655065 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655069 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655072 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655075 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655078 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655080 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655084 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655089 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655092 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.656395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655095 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655098 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655101 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655104 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655107 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655109 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655112 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655115 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655117 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655120 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655123 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655126 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655129 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655131 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655133 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655136 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655138 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655142 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655146 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.656876 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655148 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655910 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655917 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655920 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655924 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655927 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655930 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655933 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655935 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655938 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655941 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655943 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655946 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655949 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655952 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655955 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655957 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655961 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655965 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.657395 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655968 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655971 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655973 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655976 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655979 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655981 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655984 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655987 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655989 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655992 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655994 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655997 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.655999 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656002 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656004 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656007 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656009 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656012 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656014 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656016 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.657869 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656020 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656022 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656025 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656027 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656030 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656033 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656035 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656038 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656041 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656043 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656047 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656049 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656052 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656055 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656057 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656060 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656062 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656065 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656067 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.658408 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656071 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656075 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656079 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656083 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656086 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656090 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656092 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656095 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656098 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656101 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656103 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656106 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656109 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656111 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656115 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656118 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656120 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656123 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656125 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.658879 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656128 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656131 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656133 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656136 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656138 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656141 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656143 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656146 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656148 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656151 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656153 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656156 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656158 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656161 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656163 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656167 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656170 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656172 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656175 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656177 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.659359 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656180 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656183 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656187 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656578 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656584 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656587 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656590 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656593 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656598 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656602 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656606 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656610 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656613 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656616 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656618 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656621 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656624 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656626 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656629 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.659883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656631 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656634 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656636 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656639 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656641 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656644 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656646 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656649 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656652 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656654 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656657 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656659 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656662 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656665 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656667 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656670 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656672 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656675 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656677 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656680 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.660347 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656682 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656687 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656689 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656692 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656694 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656696 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656699 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656701 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656704 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656706 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656709 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656711 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656714 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656716 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656719 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656722 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656724 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656727 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656729 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.660830 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656732 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656734 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656738 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656741 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656744 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656747 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656750 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656753 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656756 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656759 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656762 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656765 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656767 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656771 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656773 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656776 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656779 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656782 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656785 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.661302 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656787 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656790 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656792 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656795 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656799 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656801 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656804 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656806 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656809 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656811 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656814 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656816 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656818 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656821 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656823 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656826 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656828 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656831 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656833 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656836 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.661750 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656838 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656841 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656843 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656846 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.656848 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656926 2573 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656969 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656976 2573 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656980 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656984 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656987 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656991 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.656995 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657001 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657004 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657007 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657011 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657014 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657017 2573 flags.go:64] FLAG: --cgroup-root="" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657020 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657023 2573 flags.go:64] FLAG: --client-ca-file="" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657025 2573 flags.go:64] FLAG: --cloud-provider="external" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657028 2573 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 01:30:53.662228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657032 2573 flags.go:64] FLAG: --cluster-domain="" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657035 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657037 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657040 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657043 2573 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657047 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657050 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657053 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657056 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657059 2573 flags.go:64] FLAG: --contention-profiling="false" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657062 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657065 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657068 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657071 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657078 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657081 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657084 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657087 2573 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657090 2573 flags.go:64] FLAG: --enable-server="true" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657093 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657098 2573 flags.go:64] FLAG: --event-burst="100" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657101 2573 flags.go:64] FLAG: --event-qps="50" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657106 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657109 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 01:30:53.662788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657112 2573 flags.go:64] FLAG: --eviction-hard="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657115 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657118 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657121 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657124 2573 flags.go:64] FLAG: --eviction-soft="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657127 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657130 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657132 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657135 2573 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657138 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657141 2573 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657144 2573 flags.go:64] FLAG: --feature-gates="" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657147 2573 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657150 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657153 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657156 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657161 2573 flags.go:64] FLAG: --healthz-port="10248" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657164 2573 flags.go:64] FLAG: --help="false" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657167 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657170 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657173 2573 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657176 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657179 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 01:30:53.663387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657182 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657185 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657188 2573 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657191 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657193 2573 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657197 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657200 2573 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657204 2573 flags.go:64] FLAG: --kube-reserved="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657207 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657209 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657212 2573 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657215 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657218 2573 flags.go:64] FLAG: --lock-file="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657221 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657223 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657226 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657231 2573 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657234 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657237 2573 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657240 2573 flags.go:64] FLAG: --logging-format="text" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657242 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657246 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657263 2573 flags.go:64] FLAG: --manifest-url="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657266 2573 flags.go:64] FLAG: --manifest-url-header="" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657271 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 01:30:53.663982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657275 2573 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657279 2573 flags.go:64] FLAG: --max-pods="110" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657282 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657285 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657288 2573 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657291 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657294 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657296 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657299 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657307 2573 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657310 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657313 2573 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657315 2573 flags.go:64] FLAG: --pod-cidr="" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657319 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657326 2573 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657329 2573 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657332 2573 flags.go:64] FLAG: --pods-per-core="0" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657335 2573 flags.go:64] FLAG: --port="10250" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657337 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657340 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e399c04d16648a1c" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657343 2573 flags.go:64] FLAG: --qos-reserved="" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657346 2573 flags.go:64] FLAG: --read-only-port="10255" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657350 2573 flags.go:64] FLAG: --register-node="true" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657352 2573 flags.go:64] FLAG: --register-with-taints="" Sep 14 01:30:53.664600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657356 2573 flags.go:64] FLAG: --registry-burst="10" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657359 2573 flags.go:64] FLAG: --registry-qps="5" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657362 2573 flags.go:64] FLAG: --reserved-cpus="" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657364 2573 flags.go:64] FLAG: --reserved-memory="" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657368 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657371 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657373 2573 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657376 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657379 2573 flags.go:64] FLAG: --runonce="false" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657383 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657386 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657389 2573 flags.go:64] FLAG: --seccomp-default="false" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657392 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657395 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657398 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657401 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657404 2573 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657407 2573 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657410 2573 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657413 2573 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657416 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657419 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657424 2573 flags.go:64] FLAG: --system-cgroups="" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657427 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657432 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 01:30:53.665279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657435 2573 flags.go:64] FLAG: --tls-cert-file="" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657438 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657442 2573 flags.go:64] FLAG: --tls-min-version="" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657444 2573 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657447 2573 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657450 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657452 2573 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657455 2573 flags.go:64] FLAG: --v="2" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657459 2573 flags.go:64] FLAG: --version="false" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657463 2573 flags.go:64] FLAG: --vmodule="" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657467 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657470 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657605 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657609 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657612 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657615 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657619 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657622 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657624 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657627 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657629 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657632 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.665883 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657635 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657639 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657643 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657647 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657651 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657654 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657657 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657662 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657665 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657668 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657672 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657675 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657678 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657681 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657684 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657687 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657690 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657693 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657696 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.666464 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657699 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657701 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657704 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657707 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657709 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657712 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657715 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657719 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657721 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657724 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657726 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657729 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657732 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657734 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657737 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657739 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657742 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657744 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657747 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657750 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.667227 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657754 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657756 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657759 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657762 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657765 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657767 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657769 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657772 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657775 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657777 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657780 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657782 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657785 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657787 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657790 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657793 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657795 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657798 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657800 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657804 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.667841 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657807 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657809 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657812 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657814 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657817 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657819 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657822 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657824 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657827 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657829 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657832 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657835 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657838 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657841 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657843 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657846 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657849 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657852 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657854 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657857 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657860 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.668448 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657862 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657865 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657867 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657870 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657873 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657875 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657878 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657881 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.657884 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.657892 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.665769 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.665783 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.665800 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.665806 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665871 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.668965 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665875 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665879 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665882 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665886 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665891 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665894 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665897 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665900 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665904 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665906 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665909 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665912 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665914 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665917 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665920 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665922 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665925 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665928 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665930 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.669362 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665932 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665935 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665937 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665940 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665944 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665947 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665950 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665953 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665955 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665959 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665961 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665964 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665967 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665969 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665972 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665974 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665977 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665979 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665982 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665984 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.669821 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665987 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665989 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665992 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665994 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665997 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.665999 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666002 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666004 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666007 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666009 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666011 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666014 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666016 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666019 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666021 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666023 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666026 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666029 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666033 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666072 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.670328 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666076 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666078 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666082 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666084 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666087 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666090 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666093 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666096 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666098 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666101 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666103 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666106 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666108 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666111 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666113 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666116 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666118 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666121 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666123 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.670814 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666126 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666129 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666131 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666134 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666136 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666139 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666141 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666144 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666146 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666148 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666151 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666153 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666157 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666161 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666163 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666166 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666169 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666172 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666174 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.671341 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666177 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.666182 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666429 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666437 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666441 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666444 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666448 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666455 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666458 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666461 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666464 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666467 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666470 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666473 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.666476 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 01:30:53.671787 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668202 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668213 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668217 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668221 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668228 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668233 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668238 2573 feature_gate.go:328] unrecognized feature gate: Example Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668244 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668264 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668270 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668274 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668279 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668290 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668294 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668299 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668303 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668307 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668312 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668316 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668321 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 01:30:53.672157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668325 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668329 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668334 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668338 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668342 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668351 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668354 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668359 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668363 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668367 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668372 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668379 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668384 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668539 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668551 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668557 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668562 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668565 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668568 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668571 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 01:30:53.672741 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668574 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668577 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668580 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668582 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668585 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668588 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668590 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668593 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668597 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668603 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668605 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668609 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668611 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668615 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668618 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668620 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668623 2573 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668626 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668629 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 01:30:53.673213 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668631 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668634 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668637 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668639 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668643 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668647 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668651 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668655 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668658 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668661 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668664 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668666 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668669 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668672 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668674 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668677 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668679 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668682 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668684 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 01:30:53.673682 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668687 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668690 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668694 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668697 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668700 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668705 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668707 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:53.668712 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.668720 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.668842 2573 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.671314 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.672386 2573 server.go:1019] "Starting client certificate rotation" Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.672486 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 01:30:53.674131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.672520 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 01:30:53.675435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.675421 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 01:30:53.676613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.676596 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 01:30:53.690476 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.690456 2573 log.go:25] "Validated CRI v1 runtime API" Sep 14 01:30:53.696369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.696352 2573 log.go:25] "Validated CRI v1 image API" Sep 14 01:30:53.699795 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.699778 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 01:30:53.702911 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.702894 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 01:30:53.703480 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.703463 2573 fs.go:135] Filesystem UUIDs: map[144ac4a9-bbcb-44c8-b9a9-2c57f36d7624:/dev/nvme0n1p4 6456113c-3d5e-400b-92c5-84f84d3bb987:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 14 01:30:53.703529 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.703480 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 01:30:53.710804 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.710688 2573 manager.go:216] Machine: {Timestamp:2026-09-14 01:30:53.709274287 +0000 UTC m=+0.382390453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098850 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9186b38b17e53d05c3e6621977ed SystemUUID:ec2f9186-b38b-17e5-3d05-c3e6621977ed BootID:5e6929d1-760e-449e-a1e0-f87a299ff84c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f2:da:64:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f2:da:64:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:70:1b:ee:9b:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 01:30:53.710804 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.710796 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 01:30:53.710970 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.710881 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 01:30:53.714368 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.714345 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 01:30:53.714510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.714370 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-205.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 01:30:53.715228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.715218 2573 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 01:30:53.715273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.715230 2573 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 01:30:53.715273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.715238 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 01:30:53.715273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.715267 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 01:30:53.716541 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.716532 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 01:30:53.722004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.721991 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 14 01:30:53.722123 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722113 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 01:30:53.722182 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722171 2573 kubelet.go:486] "Attempting to sync node with API server" Sep 14 01:30:53.722231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722199 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 01:30:53.722289 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722237 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 01:30:53.722339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722308 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-txqjg" Sep 14 01:30:53.722339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722333 2573 kubelet.go:398] "Adding apiserver pod source" Sep 14 01:30:53.722411 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722356 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 01:30:53.722737 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722726 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 01:30:53.722772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.722743 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 01:30:53.726538 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.726520 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 01:30:53.728777 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.728761 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-txqjg" Sep 14 01:30:53.729473 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.729443 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 01:30:53.729516 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.729490 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 01:30:53.731463 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731452 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731468 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731474 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731479 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731488 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731493 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731506 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731511 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731518 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 01:30:53.731522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731523 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 01:30:53.731812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731531 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 01:30:53.731812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.731536 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 01:30:53.732421 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.732408 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 01:30:53.732421 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.732419 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 01:30:53.733644 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.733624 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-205.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 01:30:53.733717 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.733624 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 01:30:53.736172 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.736159 2573 server.go:1268] "Started kubelet" Sep 14 01:30:53.736289 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.736219 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 01:30:53.736329 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.736274 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 01:30:53.736358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.736343 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 01:30:53.736923 ip-10-0-132-205 systemd[1]: Started Kubernetes Kubelet. Sep 14 01:30:53.738435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.737885 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 01:30:53.738435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.738335 2573 server.go:310] "Adding debug handlers to kubelet server" Sep 14 01:30:53.741491 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.741465 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 01:30:53.745131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745107 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 01:30:53.745131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745125 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 01:30:53.745298 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.745180 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:53.745903 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745884 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 01:30:53.745903 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745888 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 01:30:53.746043 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745915 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 01:30:53.746043 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.745971 2573 factory.go:55] Registering systemd factory Sep 14 01:30:53.746043 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746004 2573 factory.go:223] Registration of the systemd container factory successfully Sep 14 01:30:53.746192 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746097 2573 reconstruct.go:97] "Volume reconstruction finished" Sep 14 01:30:53.746192 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746108 2573 reconciler.go:29] "Reconciler: start to sync state" Sep 14 01:30:53.746192 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746187 2573 factory.go:153] Registering CRI-O factory Sep 14 01:30:53.746364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746197 2573 factory.go:223] Registration of the crio container factory successfully Sep 14 01:30:53.746364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746245 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 01:30:53.746364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746287 2573 factory.go:103] Registering Raw factory Sep 14 01:30:53.746364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746301 2573 manager.go:1196] Started watching for new ooms in manager Sep 14 01:30:53.746702 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.746652 2573 manager.go:349] Starting recovery of all containers Sep 14 01:30:53.748118 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.748097 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:53.750269 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.750232 2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-205.ec2.internal" not found Sep 14 01:30:53.750356 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.750318 2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-205.ec2.internal\" not found" node="ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.756245 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.756225 2573 manager.go:354] Recovery completed Sep 14 01:30:53.761209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.761196 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.763645 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.763630 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.763704 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.763657 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.763704 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.763667 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.764215 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.764202 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 01:30:53.764314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.764218 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.764314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.764216 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 01:30:53.764314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.764275 2573 state_mem.go:36] "Initialized new in-memory state store" Sep 14 01:30:53.765556 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.765540 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.765596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.765568 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.765596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.765577 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.765794 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.765782 2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-205.ec2.internal" not found Sep 14 01:30:53.766614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.766603 2573 policy_none.go:49] "None policy: Start" Sep 14 01:30:53.766646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.766619 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 01:30:53.766646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.766629 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 01:30:53.768622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.768611 2573 policy_none.go:47] "Start" Sep 14 01:30:53.807811 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.807794 2573 manager.go:337] "Starting Device Plugin manager" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.807840 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.807853 2573 server.go:87] "Starting device plugin registration server" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.808159 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.808171 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.808265 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.808365 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.808375 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.808908 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.808956 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:53.830749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.824917 2573 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-132-205.ec2.internal" not found Sep 14 01:30:53.853517 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.853498 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 01:30:53.854854 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.854830 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 01:30:53.854930 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.854883 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 01:30:53.854930 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.854905 2573 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 01:30:53.855015 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.854948 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 01:30:53.857169 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.857149 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:53.909086 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.909026 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.909799 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.909783 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.909865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.909815 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.909865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.909829 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.909865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.909851 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.918779 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.918760 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.918859 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.918786 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-205.ec2.internal\": node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:53.931668 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.931647 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:53.956056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.956034 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal"] Sep 14 01:30:53.956103 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.956092 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.956813 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.956799 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.956881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.956824 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.956881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.956836 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.959187 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959176 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.959334 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.959377 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959350 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.959912 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959897 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.960001 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959927 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.960001 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959899 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.960001 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959940 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.960001 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959956 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.960001 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.959970 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.962159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.962143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.962265 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.962166 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 01:30:53.963056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.963041 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientMemory" Sep 14 01:30:53.963125 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.963067 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 01:30:53.963125 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:53.963082 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeHasSufficientPID" Sep 14 01:30:53.981825 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.981799 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-205.ec2.internal\" not found" node="ip-10-0-132-205.ec2.internal" Sep 14 01:30:53.986096 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:53.986079 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-205.ec2.internal\" not found" node="ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.032658 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.032639 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.048201 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.048182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147f4b079fff27cd4d986e86591ae20a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-205.ec2.internal\" (UID: \"147f4b079fff27cd4d986e86591ae20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.048273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.048206 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.048273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.048240 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.132997 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.132973 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.149328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149306 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.149384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.149384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147f4b079fff27cd4d986e86591ae20a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-205.ec2.internal\" (UID: \"147f4b079fff27cd4d986e86591ae20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.149451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149391 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/147f4b079fff27cd4d986e86591ae20a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-205.ec2.internal\" (UID: \"147f4b079fff27cd4d986e86591ae20a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.149451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.149451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.149402 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26d684fcb484d54bd55119fc8a5be86a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal\" (UID: \"26d684fcb484d54bd55119fc8a5be86a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.233806 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.233735 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.284303 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.284243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.289129 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.289096 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:54.334801 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.334766 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.435344 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.435299 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.535897 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.535834 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.636342 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.636315 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.672402 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.672372 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 01:30:54.672853 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.672508 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 01:30:54.672853 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.672541 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 01:30:54.734813 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.734767 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 01:25:54 +0000 UTC" deadline="2028-06-10 17:38:51.156229848 +0000 UTC" Sep 14 01:30:54.734813 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.734809 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15256h7m56.421426005s" Sep 14 01:30:54.736928 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.736900 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.800943 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:54.800876 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:54.837050 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.837003 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.938192 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:54.938147 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:54.995737 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:54.995703 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod147f4b079fff27cd4d986e86591ae20a.slice/crio-690dacc8a43e5aad000284be7788d9a1f1c199dbcba266c95a89fec8bc683db9 WatchSource:0}: Error finding container 690dacc8a43e5aad000284be7788d9a1f1c199dbcba266c95a89fec8bc683db9: Status 404 returned error can't find the container with id 690dacc8a43e5aad000284be7788d9a1f1c199dbcba266c95a89fec8bc683db9 Sep 14 01:30:54.996506 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:54.996478 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d684fcb484d54bd55119fc8a5be86a.slice/crio-c6e1e21b57a4b1adabea439ea2a18dc0f4a998e89aac97d446ebd785537dc5a7 WatchSource:0}: Error finding container c6e1e21b57a4b1adabea439ea2a18dc0f4a998e89aac97d446ebd785537dc5a7: Status 404 returned error can't find the container with id c6e1e21b57a4b1adabea439ea2a18dc0f4a998e89aac97d446ebd785537dc5a7 Sep 14 01:30:55.002277 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.002257 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:30:55.038372 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.038340 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:55.139500 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.139470 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-205.ec2.internal\" not found" Sep 14 01:30:55.195561 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.195523 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:55.245864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.245841 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" Sep 14 01:30:55.258976 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.258949 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 01:30:55.260438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.260420 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" Sep 14 01:30:55.271853 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.271834 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 01:30:55.723760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.723486 2573 apiserver.go:52] "Watching apiserver" Sep 14 01:30:55.732723 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.732376 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 01:30:55.740569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.739787 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-7szvb","openshift-dns/node-resolver-6pflr","openshift-image-registry/node-ca-kbv7x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal","openshift-multus/multus-additional-cni-plugins-jqwl7","openshift-multus/multus-vgv5x","openshift-multus/network-metrics-daemon-zflq9","kube-system/global-pull-secret-syncer-25zt6","kube-system/konnectivity-agent-mq7ns","kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal","openshift-network-diagnostics/network-check-target-gfmmt","openshift-network-operator/iptables-alerter-4st6h","openshift-ovn-kubernetes/ovnkube-node-kfb47","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw"] Sep 14 01:30:55.744531 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.744502 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.746193 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.746172 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 01:30:55.746990 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.746918 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.747867 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.747689 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwfkq\"" Sep 14 01:30:55.749180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.748284 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.749180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.748526 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 01:30:55.749180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.748727 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 01:30:55.749180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.749026 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 01:30:55.749569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.749292 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.749783 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.749762 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.750045 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.750023 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6sf82\"" Sep 14 01:30:55.750417 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.750397 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.750557 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.750539 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.750771 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.750756 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 01:30:55.751826 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.751808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.752576 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.752547 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 01:30:55.752728 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.752587 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-77946\"" Sep 14 01:30:55.752827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.752811 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.753037 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.753019 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.754183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.754162 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.754373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.754357 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.754460 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.754435 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2zwzl\"" Sep 14 01:30:55.754939 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.754914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.756291 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.756224 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 01:30:55.757429 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.757409 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.757746 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.757720 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 01:30:55.758281 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.758259 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 01:30:55.758281 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.758274 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 01:30:55.758505 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.758485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.758584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.758567 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c6w2w\"" Sep 14 01:30:55.758756 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.758740 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.759842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.759842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759835 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-systemd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759863 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-log-socket\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759921 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-script-lib\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-var-lib-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.759988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760025 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-node-log\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-hosts-file\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlp2\" (UniqueName: \"kubernetes.io/projected/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-kube-api-access-jmlp2\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-etc-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-ovn\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-config\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-env-overrides\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdsv\" (UniqueName: \"kubernetes.io/projected/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-kube-api-access-6gdsv\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-tmp-dir\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-kubelet\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760311 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-systemd-units\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-slash\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-netns\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-bin\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760490 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-netd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760624 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.760726 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:30:55.760914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.760840 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 01:30:55.761307 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.761154 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7chpb\"" Sep 14 01:30:55.764286 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.764165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.764286 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.764233 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:30:55.767737 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.767212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.770129 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.770110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.771218 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.771192 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-t7znq\"" Sep 14 01:30:55.771606 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.771589 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 01:30:55.771786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.771768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 01:30:55.774384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.774360 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 01:30:55.775048 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.775020 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4xbn9\"" Sep 14 01:30:55.775617 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.775593 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.775802 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.775785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:55.775890 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.775863 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:30:55.776751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.776730 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.779492 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.779071 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4959q" Sep 14 01:30:55.779492 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.779162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.781994 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.781973 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:30:55.782096 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.781998 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckrgq\"" Sep 14 01:30:55.782096 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.782087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 01:30:55.782398 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.782383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 01:30:55.786495 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.786468 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4959q" Sep 14 01:30:55.847547 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.847527 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 01:30:55.860720 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cni-binary-copy\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.860869 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860742 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-ovn\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.860869 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-conf\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.860869 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-ovn\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lq2\" (UniqueName: \"kubernetes.io/projected/1b81705f-d12e-42a4-a01b-31bfe4247787-kube-api-access-d4lq2\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.861035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-socket-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.861035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06893e34-b6cd-4a96-9f3e-46d379a6c4be-iptables-alerter-script\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.861035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.860978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-systemd-units\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/018a746f-5f8c-474e-94c4-894ba3d0e01d-agent-certs\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlpq\" (UniqueName: \"kubernetes.io/projected/06893e34-b6cd-4a96-9f3e-46d379a6c4be-kube-api-access-2wlpq\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861075 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-systemd-units\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-script-lib\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-var-lib-kubelet\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861161 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-socket-dir-parent\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/018a746f-5f8c-474e-94c4-894ba3d0e01d-konnectivity-ca\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.861280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-system-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861295 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-netns\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-multus-certs\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00f4c9be-6791-4072-adfe-37b3b288e7e3-host\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861410 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-etc-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdsv\" (UniqueName: \"kubernetes.io/projected/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-kube-api-access-6gdsv\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-modprobe-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861479 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-lib-modules\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-host\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-slash\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-netns\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861664 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-sys-fs\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.861693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-systemd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-conf-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5dr\" (UniqueName: \"kubernetes.io/projected/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kube-api-access-sm5dr\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861838 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861889 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-etc-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861929 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-script-lib\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861928 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-slash\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861965 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-netns\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862002 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-systemd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862028 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.861842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-var-lib-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-node-log\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-hosts-file\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862271 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-var-lib-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862329 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlp2\" (UniqueName: \"kubernetes.io/projected/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-kube-api-access-jmlp2\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862337 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-node-log\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-config\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862395 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-hosts-file\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cnibin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-multus\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-device-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-kubelet\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862532 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862560 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00f4c9be-6791-4072-adfe-37b3b288e7e3-serviceca\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862574 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysconfig\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-system-cni-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.862822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862658 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-k8s-cni-cncf-io\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-etc-kubernetes\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcgg\" (UniqueName: \"kubernetes.io/projected/8cc55a86-c227-45a3-93ba-721ba72f5346-kube-api-access-5rcgg\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06893e34-b6cd-4a96-9f3e-46d379a6c4be-host-slash\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-env-overrides\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-tmp-dir\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgc6\" (UniqueName: \"kubernetes.io/projected/00f4c9be-6791-4072-adfe-37b3b288e7e3-kube-api-access-6hgc6\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-sys\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862875 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-bin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-kubelet\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-daemon-config\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-etc-selinux\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-bin\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-netd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.862998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-tuned\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.863439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863013 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-tmp\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863041 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cnibin\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-hostroot\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcphw\" (UniqueName: \"kubernetes.io/projected/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-kube-api-access-mcphw\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-log-socket\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863128 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-kubernetes\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863142 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-systemd\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6z5\" (UniqueName: \"kubernetes.io/projected/75f8117e-a6e9-40c5-9e40-06c899f78b3b-kube-api-access-tz6z5\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863173 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-kubelet-config\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-run\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863222 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovnkube-config\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863265 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-kubelet\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863301 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.864293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-dbus\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863295 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-run-openvswitch\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863477 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-bin\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-host-cni-netd\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863565 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-registration-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-log-socket\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-tmp-dir\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.863765 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-env-overrides\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.865166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.864577 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" event={"ID":"26d684fcb484d54bd55119fc8a5be86a","Type":"ContainerStarted","Data":"c6e1e21b57a4b1adabea439ea2a18dc0f4a998e89aac97d446ebd785537dc5a7"} Sep 14 01:30:55.866635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.866606 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-ovn-node-metrics-cert\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.867119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.867094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" event={"ID":"147f4b079fff27cd4d986e86591ae20a","Type":"ContainerStarted","Data":"690dacc8a43e5aad000284be7788d9a1f1c199dbcba266c95a89fec8bc683db9"} Sep 14 01:30:55.872118 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.872093 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlp2\" (UniqueName: \"kubernetes.io/projected/5206f48c-891c-4b88-b4b2-38c4cbc06a0b-kube-api-access-jmlp2\") pod \"node-resolver-6pflr\" (UID: \"5206f48c-891c-4b88-b4b2-38c4cbc06a0b\") " pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:55.873285 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.873260 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdsv\" (UniqueName: \"kubernetes.io/projected/4a03df08-6a5b-4188-9ff8-c69b5ae0904e-kube-api-access-6gdsv\") pod \"ovnkube-node-kfb47\" (UID: \"4a03df08-6a5b-4188-9ff8-c69b5ae0904e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:55.930433 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.930405 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:55.964062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lq2\" (UniqueName: \"kubernetes.io/projected/1b81705f-d12e-42a4-a01b-31bfe4247787-kube-api-access-d4lq2\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-socket-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.964327 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964091 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06893e34-b6cd-4a96-9f3e-46d379a6c4be-iptables-alerter-script\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.964327 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964236 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-socket-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.964327 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/018a746f-5f8c-474e-94c4-894ba3d0e01d-agent-certs\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlpq\" (UniqueName: \"kubernetes.io/projected/06893e34-b6cd-4a96-9f3e-46d379a6c4be-kube-api-access-2wlpq\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-var-lib-kubelet\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-socket-dir-parent\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/018a746f-5f8c-474e-94c4-894ba3d0e01d-konnectivity-ca\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-system-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964458 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-netns\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964484 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-var-lib-kubelet\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-multus-certs\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964520 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-multus-certs\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964540 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00f4c9be-6791-4072-adfe-37b3b288e7e3-host\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-modprobe-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-lib-modules\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-socket-dir-parent\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-host\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964708 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06893e34-b6cd-4a96-9f3e-46d379a6c4be-iptables-alerter-script\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-sys-fs\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964780 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-conf-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964785 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00f4c9be-6791-4072-adfe-37b3b288e7e3-host\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5dr\" (UniqueName: \"kubernetes.io/projected/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kube-api-access-sm5dr\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964836 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cnibin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-multus\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.964981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964958 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.964985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-device-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965011 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00f4c9be-6791-4072-adfe-37b3b288e7e3-serviceca\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysconfig\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-system-cni-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965086 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965111 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965114 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-system-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-k8s-cni-cncf-io\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965159 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-netns\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965163 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-etc-kubernetes\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcgg\" (UniqueName: \"kubernetes.io/projected/8cc55a86-c227-45a3-93ba-721ba72f5346-kube-api-access-5rcgg\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06893e34-b6cd-4a96-9f3e-46d379a6c4be-host-slash\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cnibin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgc6\" (UniqueName: \"kubernetes.io/projected/00f4c9be-6791-4072-adfe-37b3b288e7e3-kube-api-access-6hgc6\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-sys\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-bin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965257 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-modprobe-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.965907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-kubelet\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-daemon-config\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-etc-selinux\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-lib-modules\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-tuned\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965382 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-host\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-tmp\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965425 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cnibin\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-hostroot\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcphw\" (UniqueName: \"kubernetes.io/projected/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-kube-api-access-mcphw\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-kubernetes\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-systemd\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965611 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-systemd\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965663 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-hostroot\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cnibin\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6z5\" (UniqueName: \"kubernetes.io/projected/75f8117e-a6e9-40c5-9e40-06c899f78b3b-kube-api-access-tz6z5\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.966922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965838 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-kubelet-config\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-run\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-dbus\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-registration-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cni-binary-copy\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-conf\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966106 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966174 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-kubernetes\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.966184 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.966255 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:56.466231773 +0000 UTC m=+3.139347933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966267 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-conf\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966399 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-sys-fs\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966430 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-conf-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-dbus\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.967792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966592 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysctl-d\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966646 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-registration-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966654 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/04aebdb6-17d1-4e99-a6cd-cf767288115f-kubelet-config\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966758 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-run-k8s-cni-cncf-io\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-multus\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.966883 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.966931 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:30:56.466915379 +0000 UTC m=+3.140031548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.966984 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-device-dir\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967055 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-sys\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967126 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-etc-kubernetes\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967192 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-cni-binary-copy\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967255 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-run\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06893e34-b6cd-4a96-9f3e-46d379a6c4be-host-slash\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967452 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00f4c9be-6791-4072-adfe-37b3b288e7e3-serviceca\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.968744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967512 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-system-cni-dir\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967576 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-sysconfig\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.965036 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/018a746f-5f8c-474e-94c4-894ba3d0e01d-konnectivity-ca\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967686 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cf4f10a-9f61-47a2-93a5-3c81858ff700-etc-selinux\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967691 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-daemon-config\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967694 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-multus-cni-dir\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967741 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-cni-bin\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967767 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f8117e-a6e9-40c5-9e40-06c899f78b3b-host-var-lib-kubelet\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.967920 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.969673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.968111 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-tmp\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.970587 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.970562 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/018a746f-5f8c-474e-94c4-894ba3d0e01d-agent-certs\") pod \"konnectivity-agent-mq7ns\" (UID: \"018a746f-5f8c-474e-94c4-894ba3d0e01d\") " pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:55.970715 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.970703 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b81705f-d12e-42a4-a01b-31bfe4247787-etc-tuned\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.973642 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.973623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lq2\" (UniqueName: \"kubernetes.io/projected/1b81705f-d12e-42a4-a01b-31bfe4247787-kube-api-access-d4lq2\") pod \"tuned-7szvb\" (UID: \"1b81705f-d12e-42a4-a01b-31bfe4247787\") " pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:55.978538 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.978312 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:55.978538 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.978361 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:55.978538 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.978379 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:55.978538 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:55.978457 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:56.478437772 +0000 UTC m=+3.151553946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:55.981387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.981361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlpq\" (UniqueName: \"kubernetes.io/projected/06893e34-b6cd-4a96-9f3e-46d379a6c4be-kube-api-access-2wlpq\") pod \"iptables-alerter-4st6h\" (UID: \"06893e34-b6cd-4a96-9f3e-46d379a6c4be\") " pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:55.982165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.982129 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6z5\" (UniqueName: \"kubernetes.io/projected/75f8117e-a6e9-40c5-9e40-06c899f78b3b-kube-api-access-tz6z5\") pod \"multus-vgv5x\" (UID: \"75f8117e-a6e9-40c5-9e40-06c899f78b3b\") " pod="openshift-multus/multus-vgv5x" Sep 14 01:30:55.983276 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.983250 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcgg\" (UniqueName: \"kubernetes.io/projected/8cc55a86-c227-45a3-93ba-721ba72f5346-kube-api-access-5rcgg\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:55.983522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.983498 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgc6\" (UniqueName: \"kubernetes.io/projected/00f4c9be-6791-4072-adfe-37b3b288e7e3-kube-api-access-6hgc6\") pod \"node-ca-kbv7x\" (UID: \"00f4c9be-6791-4072-adfe-37b3b288e7e3\") " pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:55.984014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.983996 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcphw\" (UniqueName: \"kubernetes.io/projected/ee2b440b-a820-4ec4-aaaf-83ed2d4ca264-kube-api-access-mcphw\") pod \"multus-additional-cni-plugins-jqwl7\" (UID: \"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264\") " pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:55.985438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:55.985414 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5dr\" (UniqueName: \"kubernetes.io/projected/4cf4f10a-9f61-47a2-93a5-3c81858ff700-kube-api-access-sm5dr\") pod \"aws-ebs-csi-driver-node-p98mw\" (UID: \"4cf4f10a-9f61-47a2-93a5-3c81858ff700\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:56.060407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.060374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:30:56.069502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.069474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6pflr" Sep 14 01:30:56.081938 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.081913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kbv7x" Sep 14 01:30:56.087968 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.087943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7szvb" Sep 14 01:30:56.104002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.103973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" Sep 14 01:30:56.112953 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.112929 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vgv5x" Sep 14 01:30:56.120767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.120748 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:30:56.128564 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.128547 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" Sep 14 01:30:56.136416 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.136393 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4st6h" Sep 14 01:30:56.185760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.185727 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 01:30:56.470159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.470124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:56.470364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.470196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:56.470364 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.470324 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:56.470364 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.470325 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:56.470518 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.470392 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.470372995 +0000 UTC m=+4.143489183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:56.470518 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.470412 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.470402176 +0000 UTC m=+4.143518347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:56.571270 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.571219 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:56.571431 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.571390 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:56.571431 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.571409 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:56.571431 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.571421 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:56.571589 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.571483 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:57.571463449 +0000 UTC m=+4.244579604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:56.789632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.789595 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 01:25:56 +0000 UTC" deadline="2028-05-10 13:51:48.166028809 +0000 UTC" Sep 14 01:30:56.789632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.789627 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14508h20m51.376404802s" Sep 14 01:30:56.855351 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.855317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:56.855520 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:56.855357 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:56.855520 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.855464 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:30:56.855665 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:56.855631 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:30:57.090670 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.090491 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06893e34_b6cd_4a96_9f3e_46d379a6c4be.slice/crio-78a91120f88af93294fcd6289c4bfde70b2e2b0484d05ae5175b458434d53411 WatchSource:0}: Error finding container 78a91120f88af93294fcd6289c4bfde70b2e2b0484d05ae5175b458434d53411: Status 404 returned error can't find the container with id 78a91120f88af93294fcd6289c4bfde70b2e2b0484d05ae5175b458434d53411 Sep 14 01:30:57.092244 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.092194 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf4f10a_9f61_47a2_93a5_3c81858ff700.slice/crio-00b4f93f711dcfb468ec1fa4d04e09f01259ed6a6388b4480bf6a7d34a0b3fa6 WatchSource:0}: Error finding container 00b4f93f711dcfb468ec1fa4d04e09f01259ed6a6388b4480bf6a7d34a0b3fa6: Status 404 returned error can't find the container with id 00b4f93f711dcfb468ec1fa4d04e09f01259ed6a6388b4480bf6a7d34a0b3fa6 Sep 14 01:30:57.100552 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.100464 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b81705f_d12e_42a4_a01b_31bfe4247787.slice/crio-98236402d0e763b445ac24ccdbe2b80a4890bbaa32aba6d8133d40669ea23b9e WatchSource:0}: Error finding container 98236402d0e763b445ac24ccdbe2b80a4890bbaa32aba6d8133d40669ea23b9e: Status 404 returned error can't find the container with id 98236402d0e763b445ac24ccdbe2b80a4890bbaa32aba6d8133d40669ea23b9e Sep 14 01:30:57.102793 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.102477 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f4c9be_6791_4072_adfe_37b3b288e7e3.slice/crio-e6853d21b79d111aebcb610ff39ee430b10153fa490bafc4339f4c3eb7ae85f6 WatchSource:0}: Error finding container e6853d21b79d111aebcb610ff39ee430b10153fa490bafc4339f4c3eb7ae85f6: Status 404 returned error can't find the container with id e6853d21b79d111aebcb610ff39ee430b10153fa490bafc4339f4c3eb7ae85f6 Sep 14 01:30:57.104033 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.103361 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f8117e_a6e9_40c5_9e40_06c899f78b3b.slice/crio-4cb5abcfa2387386c0258eebd84454acc70d11a6d00131d90962112c2228112b WatchSource:0}: Error finding container 4cb5abcfa2387386c0258eebd84454acc70d11a6d00131d90962112c2228112b: Status 404 returned error can't find the container with id 4cb5abcfa2387386c0258eebd84454acc70d11a6d00131d90962112c2228112b Sep 14 01:30:57.105043 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:30:57.104989 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018a746f_5f8c_474e_94c4_894ba3d0e01d.slice/crio-9f7eb5c85cf922d36c1e32fa65bf5b19c1ba6636ed745fa7c1806f5e42fbc051 WatchSource:0}: Error finding container 9f7eb5c85cf922d36c1e32fa65bf5b19c1ba6636ed745fa7c1806f5e42fbc051: Status 404 returned error can't find the container with id 9f7eb5c85cf922d36c1e32fa65bf5b19c1ba6636ed745fa7c1806f5e42fbc051 Sep 14 01:30:57.480550 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.480514 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:57.480694 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.480569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:57.480694 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.480678 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.480769 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.480686 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:57.480769 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.480733 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:30:59.480718613 +0000 UTC m=+6.153834767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:57.480769 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.480749 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:59.48074219 +0000 UTC m=+6.153858345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:57.581678 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.581632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:57.581865 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.581821 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:57.581865 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.581853 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:57.581986 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.581870 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.581986 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.581933 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:30:59.581914026 +0000 UTC m=+6.255030196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:57.790168 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.790096 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 01:25:56 +0000 UTC" deadline="2028-06-24 20:13:35.659698924 +0000 UTC" Sep 14 01:30:57.790168 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.790166 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15594h42m37.869537787s" Sep 14 01:30:57.857990 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.857943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:57.858176 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:57.858096 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:30:57.880555 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.880462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" event={"ID":"4cf4f10a-9f61-47a2-93a5-3c81858ff700","Type":"ContainerStarted","Data":"00b4f93f711dcfb468ec1fa4d04e09f01259ed6a6388b4480bf6a7d34a0b3fa6"} Sep 14 01:30:57.887859 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.887771 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgv5x" event={"ID":"75f8117e-a6e9-40c5-9e40-06c899f78b3b","Type":"ContainerStarted","Data":"4cb5abcfa2387386c0258eebd84454acc70d11a6d00131d90962112c2228112b"} Sep 14 01:30:57.909220 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.909105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mq7ns" event={"ID":"018a746f-5f8c-474e-94c4-894ba3d0e01d","Type":"ContainerStarted","Data":"9f7eb5c85cf922d36c1e32fa65bf5b19c1ba6636ed745fa7c1806f5e42fbc051"} Sep 14 01:30:57.918084 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.917960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7szvb" event={"ID":"1b81705f-d12e-42a4-a01b-31bfe4247787","Type":"ContainerStarted","Data":"98236402d0e763b445ac24ccdbe2b80a4890bbaa32aba6d8133d40669ea23b9e"} Sep 14 01:30:57.926280 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.926205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4st6h" event={"ID":"06893e34-b6cd-4a96-9f3e-46d379a6c4be","Type":"ContainerStarted","Data":"78a91120f88af93294fcd6289c4bfde70b2e2b0484d05ae5175b458434d53411"} Sep 14 01:30:57.932759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.932729 2573 generic.go:356] "Generic (PLEG): container finished" podID="26d684fcb484d54bd55119fc8a5be86a" containerID="036dffbd7ce16def43c9e57f04b65f5b965583c462f605032a4cf80b056a2d86" exitCode=0 Sep 14 01:30:57.933744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.933470 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" event={"ID":"26d684fcb484d54bd55119fc8a5be86a","Type":"ContainerDied","Data":"036dffbd7ce16def43c9e57f04b65f5b965583c462f605032a4cf80b056a2d86"} Sep 14 01:30:57.951636 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.951579 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" event={"ID":"147f4b079fff27cd4d986e86591ae20a","Type":"ContainerStarted","Data":"327a5ee5416f3f9723172af0f7135eb6e42638b758b99d8687ed50b68424bb2c"} Sep 14 01:30:57.966117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.965927 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-205.ec2.internal" podStartSLOduration=2.965907786 podStartE2EDuration="2.965907786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:30:57.964675501 +0000 UTC m=+4.637791682" watchObservedRunningTime="2026-09-14 01:30:57.965907786 +0000 UTC m=+4.639023967" Sep 14 01:30:57.966704 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.966573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerStarted","Data":"42429066d779f3ea4c9a034c8b319ac87603aeabff16fdde643fd9daff2f0ce8"} Sep 14 01:30:57.987569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.987251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kbv7x" event={"ID":"00f4c9be-6791-4072-adfe-37b3b288e7e3","Type":"ContainerStarted","Data":"e6853d21b79d111aebcb610ff39ee430b10153fa490bafc4339f4c3eb7ae85f6"} Sep 14 01:30:57.999167 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:57.999133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pflr" event={"ID":"5206f48c-891c-4b88-b4b2-38c4cbc06a0b","Type":"ContainerStarted","Data":"8efdde41716e0a607c9f889bd2a4fb54d9bd2b163ae2049840da8279b4b96a90"} Sep 14 01:30:58.002748 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:58.002614 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"ae42389a4115ed95bbd3834974e09c7f19e9f95853bb47cb0e68e89adf633624"} Sep 14 01:30:58.855385 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:58.855345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:58.855868 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:58.855492 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:30:58.855952 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:58.855910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:58.856046 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:58.856020 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:30:59.025025 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:59.024944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" event={"ID":"26d684fcb484d54bd55119fc8a5be86a","Type":"ContainerStarted","Data":"215c4fa8dd602d0b71eceea221649a4dc3093fdd14fd41d0b5d825a4d1f6719b"} Sep 14 01:30:59.497539 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:59.497248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:30:59.497727 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:59.497576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:59.497794 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.497744 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:59.497859 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.497796 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:31:03.497783274 +0000 UTC m=+10.170899443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:30:59.498143 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.498125 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:59.498220 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.498180 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:03.498164578 +0000 UTC m=+10.171280748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:30:59.598682 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:59.598019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:30:59.598682 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.598188 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:30:59.598682 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.598209 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:30:59.598682 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.598223 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:59.598682 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.598284 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:03.598264844 +0000 UTC m=+10.271381016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:30:59.866507 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:30:59.866430 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:30:59.866962 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:30:59.866561 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:00.855420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:00.855386 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:00.855628 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:00.855495 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:00.855824 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:00.855808 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:00.855906 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:00.855890 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:01.855730 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:01.855699 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:01.856169 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:01.855832 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:02.855364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:02.855326 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:02.855562 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:02.855447 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:02.855935 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:02.855793 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:02.855935 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:02.855891 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:03.533637 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:03.533598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:03.533815 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:03.533655 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:03.533815 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.533763 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:03.533815 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.533816 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:31:11.533803343 +0000 UTC m=+18.206919509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:03.534400 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.534186 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:03.534400 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.534241 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:11.534224652 +0000 UTC m=+18.207340835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:03.634518 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:03.634403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:03.634691 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.634561 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:03.634691 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.634589 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:03.634691 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.634603 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:03.634691 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.634667 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:11.634647853 +0000 UTC m=+18.307764009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:03.859374 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:03.859305 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:03.859756 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:03.859413 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:04.855870 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:04.855831 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:04.856077 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:04.855849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:04.856077 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:04.855968 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:04.856077 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:04.856054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:05.855936 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:05.855860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:05.856411 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:05.856116 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:06.855485 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:06.855456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:06.855629 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:06.855496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:06.855629 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:06.855570 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:06.855738 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:06.855718 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:07.855766 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:07.855733 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:07.856201 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:07.855881 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:08.855401 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:08.855353 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:08.855559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:08.855359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:08.856157 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:08.856135 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:08.856535 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:08.855828 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:09.855966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:09.855915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:09.856135 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:09.856057 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:10.855830 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:10.855792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:10.856376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:10.855806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:10.856376 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:10.855930 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:10.856376 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:10.855985 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:11.589920 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:11.589876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:11.590081 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:11.589962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:11.590081 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.590015 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:11.590081 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.590071 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:11.590197 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.590089 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:27.590071776 +0000 UTC m=+34.263187947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:11.590197 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.590123 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:31:27.590107024 +0000 UTC m=+34.263223191 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:11.690465 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:11.690424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:11.690614 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.690578 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:11.690614 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.690603 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:11.690731 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.690617 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:11.690731 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.690678 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:27.690659563 +0000 UTC m=+34.363775734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:11.855977 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:11.855913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:11.856355 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:11.856022 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:12.855195 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:12.855163 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:12.855195 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:12.855179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:12.855406 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:12.855297 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:12.855475 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:12.855424 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:13.856222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:13.856189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:13.856707 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:13.856341 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:14.855137 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:14.855100 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:14.855328 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:14.855221 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:14.855673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:14.855650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:14.855813 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:14.855754 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:15.856468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:15.856231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:15.857186 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:15.856602 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:16.067217 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.067181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vgv5x" event={"ID":"75f8117e-a6e9-40c5-9e40-06c899f78b3b","Type":"ContainerStarted","Data":"d9307b7e50ad7371fa50f01d81650377fd73ed302a31fce5c4dc930a956c9a47"} Sep 14 01:31:16.068653 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.068629 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mq7ns" event={"ID":"018a746f-5f8c-474e-94c4-894ba3d0e01d","Type":"ContainerStarted","Data":"faa62a08578cbdd774cee64dafd78bda50ebea289b4848504ae140631917ae94"} Sep 14 01:31:16.070207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.070182 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7szvb" event={"ID":"1b81705f-d12e-42a4-a01b-31bfe4247787","Type":"ContainerStarted","Data":"3cdafba5bbf98388943d2b698ad4d9140d4dcca2df51a804d1f0d3a8014a8ea7"} Sep 14 01:31:16.071484 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.071452 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="0bc6a206948dd31a81118b03f58184b27fd888184beb584c9bd671d7a3163214" exitCode=0 Sep 14 01:31:16.071583 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.071530 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"0bc6a206948dd31a81118b03f58184b27fd888184beb584c9bd671d7a3163214"} Sep 14 01:31:16.072963 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.072945 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kbv7x" event={"ID":"00f4c9be-6791-4072-adfe-37b3b288e7e3","Type":"ContainerStarted","Data":"e5121267efd2af1f30104ff415c09586ebc62077df779ecfb0f5e75709c26954"} Sep 14 01:31:16.074286 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.074264 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6pflr" event={"ID":"5206f48c-891c-4b88-b4b2-38c4cbc06a0b","Type":"ContainerStarted","Data":"6795fcd47a527433b055cec466ff0d2d217b778c5c5976a00e5ea52db18a1bd2"} Sep 14 01:31:16.077022 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.076998 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"8f2ab00ee2c3b617c766753f7425f8298d9973207b80faaa4d054297a2796cd0"} Sep 14 01:31:16.077108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.077049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"ffbfc2c71d0c966b514f6ff0b10f6531523fc9fbccbd3b2aac8c23567da60f31"} Sep 14 01:31:16.077108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.077087 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"a407e9866f3813105a89714ddbe2361d90e053e72eb49de0a3afd12d3bd4c805"} Sep 14 01:31:16.077181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.077135 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"e1d84875e1548609a88942a7e29277a6ce0fa352b7a8054f5cb2d1f40799cb2f"} Sep 14 01:31:16.077220 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.077181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"80c1d256b98d4758d208fa69d594234bf365893348197ff3c5deebfb61b0650e"} Sep 14 01:31:16.078598 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.078581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" event={"ID":"4cf4f10a-9f61-47a2-93a5-3c81858ff700","Type":"ContainerStarted","Data":"c32a85291d238d6b2f203e66f815a2c25733b11260f011a9797bcc7dc3993bb9"} Sep 14 01:31:16.083034 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.082986 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-205.ec2.internal" podStartSLOduration=21.08297162 podStartE2EDuration="21.08297162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:30:59.040439346 +0000 UTC m=+5.713555526" watchObservedRunningTime="2026-09-14 01:31:16.08297162 +0000 UTC m=+22.756087797" Sep 14 01:31:16.083150 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.083130 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vgv5x" podStartSLOduration=3.603747289 podStartE2EDuration="22.083124248s" totalImagesPullingTime="18.479376959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.082638583 +0000 UTC m=+22.755754758" watchObservedRunningTime="2026-09-14 01:31:16.083124248 +0000 UTC m=+22.756240423" Sep 14 01:31:16.095780 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.095736 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6pflr" podStartSLOduration=3.6606176980000003 podStartE2EDuration="22.095717878s" totalImagesPullingTime="18.43510018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.095151747 +0000 UTC m=+22.768267921" watchObservedRunningTime="2026-09-14 01:31:16.095717878 +0000 UTC m=+22.768834246" Sep 14 01:31:16.107709 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.107675 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kbv7x" podStartSLOduration=3.690949435 podStartE2EDuration="22.107662506s" totalImagesPullingTime="18.416713071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.107321489 +0000 UTC m=+22.780437663" watchObservedRunningTime="2026-09-14 01:31:16.107662506 +0000 UTC m=+22.780778681" Sep 14 01:31:16.127565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.127528 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7szvb" podStartSLOduration=3.661069375 podStartE2EDuration="22.127516552s" totalImagesPullingTime="18.466447177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.127228631 +0000 UTC m=+22.800344806" watchObservedRunningTime="2026-09-14 01:31:16.127516552 +0000 UTC m=+22.800632726" Sep 14 01:31:16.161379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.161315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mq7ns" podStartSLOduration=3.747008333 podStartE2EDuration="22.161300879s" totalImagesPullingTime="18.414292546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:16.160558395 +0000 UTC m=+22.833674576" watchObservedRunningTime="2026-09-14 01:31:16.161300879 +0000 UTC m=+22.834417053" Sep 14 01:31:16.231464 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.231433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mq7ns_018a746f-5f8c-474e-94c4-894ba3d0e01d/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:16.313981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.313874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mq7ns_018a746f-5f8c-474e-94c4-894ba3d0e01d/konnectivity-agent/0.log" limit=1024 Sep 14 01:31:16.855490 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.855466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:16.855646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:16.855466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:16.855646 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:16.855601 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:16.855646 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:16.855641 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:17.084024 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.083578 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4st6h" event={"ID":"06893e34-b6cd-4a96-9f3e-46d379a6c4be","Type":"ContainerStarted","Data":"f56539001af1ffb8001a00307b5534d36f98762da757a014b6117e1320952b41"} Sep 14 01:31:17.087680 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.087645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"cfd9d1e5ec661ad4a6a9f453f6256f608e2f9a3ef0b34a684539aa268354b5c9"} Sep 14 01:31:17.099700 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.099649 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4st6h" podStartSLOduration=4.662601997 podStartE2EDuration="23.099631597s" totalImagesPullingTime="18.4370296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:17.099441207 +0000 UTC m=+23.772557381" watchObservedRunningTime="2026-09-14 01:31:17.099631597 +0000 UTC m=+23.772747773" Sep 14 01:31:17.165650 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.165626 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 01:31:17.819020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.818314 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T01:31:17.165642143Z","UUID":"a3a4ae1f-54e3-44bb-b0af-6c191045495e","Handler":null,"Name":"","Endpoint":""} Sep 14 01:31:17.820044 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.820011 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 01:31:17.820177 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.820053 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 01:31:17.855739 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:17.855708 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:17.855915 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:17.855841 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:18.092751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:18.092676 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" event={"ID":"4cf4f10a-9f61-47a2-93a5-3c81858ff700","Type":"ContainerStarted","Data":"c192bcbe92865bf0633f0ae22ceebe1b5eca0951891b046e19e9e39776ebe882"} Sep 14 01:31:18.855802 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:18.855770 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:18.855964 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:18.855813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:18.855964 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:18.855922 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:18.856078 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:18.856049 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:19.098737 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:19.098701 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"ab217e1b24837870dfa6846d5183e89c62fbcc706e0f61f395147f0994c7fbff"} Sep 14 01:31:19.104062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:19.104032 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" event={"ID":"4cf4f10a-9f61-47a2-93a5-3c81858ff700","Type":"ContainerStarted","Data":"5dcba08cc797123f8067437feb76c33e426ea0b9894eeee8f60064041c21927f"} Sep 14 01:31:19.120070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:19.119989 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p98mw" podStartSLOduration=4.25526678 podStartE2EDuration="25.119976491s" totalImagesPullingTime="20.864709711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:19.119582326 +0000 UTC m=+25.792698501" watchObservedRunningTime="2026-09-14 01:31:19.119976491 +0000 UTC m=+25.793092665" Sep 14 01:31:19.855382 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:19.855346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:19.855591 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:19.855492 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:20.688912 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:20.688878 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:31:20.689548 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:20.689530 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:31:20.855501 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:20.855466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:20.855669 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:20.855482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:20.855669 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:20.855593 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:20.855796 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:20.855684 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:21.108781 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:21.108713 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:31:21.109127 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:21.109112 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mq7ns" Sep 14 01:31:21.855813 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:21.855609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:21.856323 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:21.855856 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:22.112648 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.112571 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="d70b8a77cd78cbc940f63891f72be8aa5216d6a0f4cf8714db8920677a90654a" exitCode=0 Sep 14 01:31:22.112778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.112648 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"d70b8a77cd78cbc940f63891f72be8aa5216d6a0f4cf8714db8920677a90654a"} Sep 14 01:31:22.116332 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.116310 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" event={"ID":"4a03df08-6a5b-4188-9ff8-c69b5ae0904e","Type":"ContainerStarted","Data":"ef864f2655af4072bcfd1b08a02c8eb07105bc4d7a025c35671c19108fc87985"} Sep 14 01:31:22.116630 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.116610 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:22.135075 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.135048 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:22.155325 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.155287 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" podStartSLOduration=9.631659351 podStartE2EDuration="28.155274264s" totalImagesPullingTime="18.523614913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:22.155134849 +0000 UTC m=+28.828251034" watchObservedRunningTime="2026-09-14 01:31:22.155274264 +0000 UTC m=+28.828390435" Sep 14 01:31:22.855896 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.855863 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:22.856653 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:22.856042 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:22.856653 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:22.856421 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:22.856653 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:22.856525 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:23.121244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.121128 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="4a3cc16ed6473a0cecc4829b5131ce951083fd041b30dfcc7995d380c2e147e5" exitCode=0 Sep 14 01:31:23.121244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.121203 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"4a3cc16ed6473a0cecc4829b5131ce951083fd041b30dfcc7995d380c2e147e5"} Sep 14 01:31:23.122479 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.122459 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:23.123221 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.123201 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:23.133228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.133208 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfmmt"] Sep 14 01:31:23.133328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.133302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:23.133397 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:23.133382 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:23.137321 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.137300 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zflq9"] Sep 14 01:31:23.137415 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.137354 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-25zt6"] Sep 14 01:31:23.137467 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.137431 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:23.137529 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:23.137510 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:23.137583 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.137574 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:23.137736 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:23.137641 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:23.142819 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:23.142793 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:24.126309 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:24.126202 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="1fc153a8a745282381ddb38a6745fe537bc92a4447cd74a3b097e2178378bda9" exitCode=0 Sep 14 01:31:24.126309 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:24.126290 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"1fc153a8a745282381ddb38a6745fe537bc92a4447cd74a3b097e2178378bda9"} Sep 14 01:31:24.856010 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:24.855980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:24.856151 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:24.856020 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:24.856151 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:24.855980 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:24.856151 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:24.856106 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:24.856299 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:24.856222 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:24.856380 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:24.856349 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:26.855936 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:26.855894 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:26.856627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:26.855892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:26.856627 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:26.856032 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfmmt" podUID="adb2052f-098e-4e97-b7e1-62c30492c194" Sep 14 01:31:26.856627 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:26.856115 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-25zt6" podUID="04aebdb6-17d1-4e99-a6cd-cf767288115f" Sep 14 01:31:26.856627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:26.855892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:26.856627 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:26.856277 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:31:27.610278 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:27.610237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:27.610471 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:27.610305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:27.610471 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.610408 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:27.610471 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.610434 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:27.610619 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.610485 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:59.610463507 +0000 UTC m=+66.283579660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 01:31:27.610619 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.610505 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret podName:04aebdb6-17d1-4e99-a6cd-cf767288115f nodeName:}" failed. No retries permitted until 2026-09-14 01:31:59.610495774 +0000 UTC m=+66.283611927 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret") pod "global-pull-secret-syncer-25zt6" (UID: "04aebdb6-17d1-4e99-a6cd-cf767288115f") : object "kube-system"/"original-pull-secret" not registered Sep 14 01:31:27.710938 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:27.710904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:27.711098 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.711081 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 01:31:27.711153 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.711104 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 01:31:27.711153 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.711116 2573 projected.go:196] Error preparing data for projected volume kube-api-access-wsfqg for pod openshift-network-diagnostics/network-check-target-gfmmt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:27.711226 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:27.711164 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg podName:adb2052f-098e-4e97-b7e1-62c30492c194 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:59.711151099 +0000 UTC m=+66.384267257 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wsfqg" (UniqueName: "kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg") pod "network-check-target-gfmmt" (UID: "adb2052f-098e-4e97-b7e1-62c30492c194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 01:31:28.612695 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.612664 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-205.ec2.internal" event="NodeReady" Sep 14 01:31:28.613050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.612786 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 01:31:28.650544 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.650518 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vwkwr"] Sep 14 01:31:28.667980 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.667954 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nk9kd"] Sep 14 01:31:28.668101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.668075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.670471 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.670451 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jfmw6\"" Sep 14 01:31:28.670471 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.670466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 01:31:28.670732 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.670719 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 01:31:28.690345 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.690324 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vwkwr"] Sep 14 01:31:28.690448 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.690367 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nk9kd"] Sep 14 01:31:28.690502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.690469 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.692793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.692772 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 01:31:28.692890 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.692772 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 01:31:28.692890 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.692779 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tbbp2\"" Sep 14 01:31:28.692890 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.692838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 01:31:28.819228 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf444def-59a8-44d1-8ac4-8240cdec1922-tmp-dir\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.819379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf444def-59a8-44d1-8ac4-8240cdec1922-config-volume\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.819379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.819379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819371 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5k7\" (UniqueName: \"kubernetes.io/projected/36788e14-cf89-4938-8495-fafa3349508a-kube-api-access-dk5k7\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.819480 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbv9\" (UniqueName: \"kubernetes.io/projected/bf444def-59a8-44d1-8ac4-8240cdec1922-kube-api-access-llbv9\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.819480 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.819421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36788e14-cf89-4938-8495-fafa3349508a-cert\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.856128 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.856093 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:28.856128 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.856118 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:28.856269 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.856215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:28.859053 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859030 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 01:31:28.859053 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859050 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 01:31:28.859242 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859053 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 01:31:28.859242 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859197 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tck5c\"" Sep 14 01:31:28.859358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859306 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 01:31:28.859358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.859317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gv4zq\"" Sep 14 01:31:28.920314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36788e14-cf89-4938-8495-fafa3349508a-cert\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.920314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf444def-59a8-44d1-8ac4-8240cdec1922-tmp-dir\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.920483 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf444def-59a8-44d1-8ac4-8240cdec1922-config-volume\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.920574 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.920574 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5k7\" (UniqueName: \"kubernetes.io/projected/36788e14-cf89-4938-8495-fafa3349508a-kube-api-access-dk5k7\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.920666 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf444def-59a8-44d1-8ac4-8240cdec1922-tmp-dir\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.920666 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920615 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbv9\" (UniqueName: \"kubernetes.io/projected/bf444def-59a8-44d1-8ac4-8240cdec1922-kube-api-access-llbv9\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.920666 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:28.920622 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:28.920764 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:28.920688 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:29.420669968 +0000 UTC m=+36.093786135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:28.920980 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.920960 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf444def-59a8-44d1-8ac4-8240cdec1922-config-volume\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.924544 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.924528 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36788e14-cf89-4938-8495-fafa3349508a-cert\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:28.929034 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.929009 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbv9\" (UniqueName: \"kubernetes.io/projected/bf444def-59a8-44d1-8ac4-8240cdec1922-kube-api-access-llbv9\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:28.929122 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:28.929104 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5k7\" (UniqueName: \"kubernetes.io/projected/36788e14-cf89-4938-8495-fafa3349508a-kube-api-access-dk5k7\") pod \"ingress-canary-nk9kd\" (UID: \"36788e14-cf89-4938-8495-fafa3349508a\") " pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:29.001141 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:29.001121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nk9kd" Sep 14 01:31:29.141372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:29.141187 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerStarted","Data":"095e977622e7746b4888073dc990cdccd0eef486cc1a9a83d80f2a71bdccafb6"} Sep 14 01:31:29.330472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:29.330442 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nk9kd"] Sep 14 01:31:29.333164 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:31:29.333138 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36788e14_cf89_4938_8495_fafa3349508a.slice/crio-ade2e07e18e424f91dc239c10d2d31ce34f0c44491a3d96210e4c66cf943edca WatchSource:0}: Error finding container ade2e07e18e424f91dc239c10d2d31ce34f0c44491a3d96210e4c66cf943edca: Status 404 returned error can't find the container with id ade2e07e18e424f91dc239c10d2d31ce34f0c44491a3d96210e4c66cf943edca Sep 14 01:31:29.425955 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:29.425928 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:29.426050 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:29.426039 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:29.426101 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:29.426093 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:30.426079561 +0000 UTC m=+37.099195727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:30.147202 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:30.147159 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"095e977622e7746b4888073dc990cdccd0eef486cc1a9a83d80f2a71bdccafb6"} Sep 14 01:31:30.147859 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:30.147074 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="095e977622e7746b4888073dc990cdccd0eef486cc1a9a83d80f2a71bdccafb6" exitCode=0 Sep 14 01:31:30.149168 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:30.149148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nk9kd" event={"ID":"36788e14-cf89-4938-8495-fafa3349508a","Type":"ContainerStarted","Data":"ade2e07e18e424f91dc239c10d2d31ce34f0c44491a3d96210e4c66cf943edca"} Sep 14 01:31:30.433809 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:30.433736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:30.433937 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:30.433918 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:30.434009 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:30.433998 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:32.433976547 +0000 UTC m=+39.107092717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:31.161981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:31.161900 2573 generic.go:356] "Generic (PLEG): container finished" podID="ee2b440b-a820-4ec4-aaaf-83ed2d4ca264" containerID="2ae1d6a774a9408b1f50a97a229b85c3b73d66fb8196d453a53bf7729702a30d" exitCode=0 Sep 14 01:31:31.161981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:31.161956 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerDied","Data":"2ae1d6a774a9408b1f50a97a229b85c3b73d66fb8196d453a53bf7729702a30d"} Sep 14 01:31:32.168675 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:32.168635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" event={"ID":"ee2b440b-a820-4ec4-aaaf-83ed2d4ca264","Type":"ContainerStarted","Data":"8b7e8a12df4fee02bb0b56ced45019735ae4e94393ccc7a3cb1f961d41708c37"} Sep 14 01:31:32.170295 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:32.170247 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nk9kd" event={"ID":"36788e14-cf89-4938-8495-fafa3349508a","Type":"ContainerStarted","Data":"6d84441212b96932c4fd300db729ae69daf65d2276671e46da931b17071d9f96"} Sep 14 01:31:32.189062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:32.189016 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqwl7" podStartSLOduration=8.245188556 podStartE2EDuration="38.189002268s" totalImagesPullingTime="29.943813712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:32.187902161 +0000 UTC m=+38.861018335" watchObservedRunningTime="2026-09-14 01:31:32.189002268 +0000 UTC m=+38.862118425" Sep 14 01:31:32.201404 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:32.201364 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nk9kd" podStartSLOduration=1.5784544299999999 podStartE2EDuration="4.201351249s" totalImagesPullingTime="2.622896819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:31:32.200473101 +0000 UTC m=+38.873589276" watchObservedRunningTime="2026-09-14 01:31:32.201351249 +0000 UTC m=+38.874467423" Sep 14 01:31:32.451372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:32.451338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:32.451540 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:32.451460 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:32.451540 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:32.451538 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:36.451525314 +0000 UTC m=+43.124641467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:36.482408 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:36.482367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:36.482766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:36.482516 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:36.482766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:36.482576 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:31:44.482562633 +0000 UTC m=+51.155678798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:44.538419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:44.538382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:31:44.538819 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:44.538533 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:31:44.538819 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:44.538599 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:32:00.538582082 +0000 UTC m=+67.211698251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:31:55.139273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:55.139228 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfb47" Sep 14 01:31:59.641396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.641358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:31:59.641766 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.641412 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:59.644033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.644016 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 01:31:59.644204 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.644182 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 01:31:59.652234 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:59.652215 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:31:59.652349 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:31:59.652308 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:33:03.652285291 +0000 UTC m=+130.325401444 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : secret "metrics-daemon-secret" not found Sep 14 01:31:59.655481 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.655466 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/04aebdb6-17d1-4e99-a6cd-cf767288115f-original-pull-secret\") pod \"global-pull-secret-syncer-25zt6\" (UID: \"04aebdb6-17d1-4e99-a6cd-cf767288115f\") " pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:59.742397 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.742369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:59.745626 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.745610 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 01:31:59.756050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.756036 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 01:31:59.766646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.766628 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsfqg\" (UniqueName: \"kubernetes.io/projected/adb2052f-098e-4e97-b7e1-62c30492c194-kube-api-access-wsfqg\") pod \"network-check-target-gfmmt\" (UID: \"adb2052f-098e-4e97-b7e1-62c30492c194\") " pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:31:59.771391 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.771373 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-25zt6" Sep 14 01:31:59.777645 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.777629 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tck5c\"" Sep 14 01:31:59.786458 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:31:59.786445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:32:00.068634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:00.068562 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-25zt6"] Sep 14 01:32:00.071574 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:32:00.071549 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04aebdb6_17d1_4e99_a6cd_cf767288115f.slice/crio-9ccfc19cf2c40a71e3d5c6a4f113360a1010a6a1de3152a7a062d217e664f60a WatchSource:0}: Error finding container 9ccfc19cf2c40a71e3d5c6a4f113360a1010a6a1de3152a7a062d217e664f60a: Status 404 returned error can't find the container with id 9ccfc19cf2c40a71e3d5c6a4f113360a1010a6a1de3152a7a062d217e664f60a Sep 14 01:32:00.094467 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:00.094442 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfmmt"] Sep 14 01:32:00.097233 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:32:00.097212 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb2052f_098e_4e97_b7e1_62c30492c194.slice/crio-9fc1739e2263e296f31e5b34de57bd9a360921a963eb56b4fed79c297e703013 WatchSource:0}: Error finding container 9fc1739e2263e296f31e5b34de57bd9a360921a963eb56b4fed79c297e703013: Status 404 returned error can't find the container with id 9fc1739e2263e296f31e5b34de57bd9a360921a963eb56b4fed79c297e703013 Sep 14 01:32:00.235379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:00.235343 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfmmt" event={"ID":"adb2052f-098e-4e97-b7e1-62c30492c194","Type":"ContainerStarted","Data":"9fc1739e2263e296f31e5b34de57bd9a360921a963eb56b4fed79c297e703013"} Sep 14 01:32:00.236514 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:00.236487 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-25zt6" event={"ID":"04aebdb6-17d1-4e99-a6cd-cf767288115f","Type":"ContainerStarted","Data":"9ccfc19cf2c40a71e3d5c6a4f113360a1010a6a1de3152a7a062d217e664f60a"} Sep 14 01:32:00.547911 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:00.547872 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:32:00.548123 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:32:00.548046 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:32:00.548194 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:32:00.548129 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:32:32.548107158 +0000 UTC m=+99.221223325 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:32:05.251681 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:05.251644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfmmt" event={"ID":"adb2052f-098e-4e97-b7e1-62c30492c194","Type":"ContainerStarted","Data":"ef211aa536520c87eda1ae063ee2969646930210726d3e18171c2fd8385905db"} Sep 14 01:32:05.252179 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:05.251751 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:32:05.253226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:05.253187 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-25zt6" event={"ID":"04aebdb6-17d1-4e99-a6cd-cf767288115f","Type":"ContainerStarted","Data":"9ca663522650f3104be4905f97ad7d560014e3e007be7abbce5401b5600615d7"} Sep 14 01:32:05.265964 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:05.265912 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gfmmt" podStartSLOduration=66.672368733 podStartE2EDuration="1m11.265897614s" totalImagesPullingTime="4.593528881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:05.265858415 +0000 UTC m=+71.938974589" watchObservedRunningTime="2026-09-14 01:32:05.265897614 +0000 UTC m=+71.939013786" Sep 14 01:32:05.282628 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:05.282584 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-25zt6" podStartSLOduration=66.658372982 podStartE2EDuration="1m11.282566383s" totalImagesPullingTime="4.624193401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:05.281717795 +0000 UTC m=+71.954833969" watchObservedRunningTime="2026-09-14 01:32:05.282566383 +0000 UTC m=+71.955682558" Sep 14 01:32:32.566487 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:32.566347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:32:32.566952 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:32:32.566502 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:32:32.566952 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:32:32.566581 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:33:36.566564782 +0000 UTC m=+163.239680935 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:32:36.117265 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.117232 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2"] Sep 14 01:32:36.146803 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.146781 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2"] Sep 14 01:32:36.146941 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.146867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.149345 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.149321 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 01:32:36.150603 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.150581 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 01:32:36.150773 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.150684 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 01:32:36.150916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.150900 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 14 01:32:36.151836 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.151817 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm"] Sep 14 01:32:36.172528 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.172509 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm"] Sep 14 01:32:36.172616 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.172590 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.175014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.174996 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 14 01:32:36.175105 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.175028 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 14 01:32:36.175105 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.175084 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 14 01:32:36.175105 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.175028 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 14 01:32:36.190244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.190217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0eacb30d-a06a-468d-96f9-ff39199c9f76-klusterlet-config\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.190364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.190273 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzzn\" (UniqueName: \"kubernetes.io/projected/0eacb30d-a06a-468d-96f9-ff39199c9f76-kube-api-access-fkzzn\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.190424 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.190379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eacb30d-a06a-468d-96f9-ff39199c9f76-tmp\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.259056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.259027 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gfmmt" Sep 14 01:32:36.290868 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.290843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.290885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e9f7ed-4d64-482a-9beb-418dd587dda7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.290903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.290969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291028 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eacb30d-a06a-468d-96f9-ff39199c9f76-tmp\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.291181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291090 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0eacb30d-a06a-468d-96f9-ff39199c9f76-klusterlet-config\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.291181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48fz\" (UniqueName: \"kubernetes.io/projected/58e9f7ed-4d64-482a-9beb-418dd587dda7-kube-api-access-r48fz\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.291410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzzn\" (UniqueName: \"kubernetes.io/projected/0eacb30d-a06a-468d-96f9-ff39199c9f76-kube-api-access-fkzzn\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.291509 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.291491 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0eacb30d-a06a-468d-96f9-ff39199c9f76-tmp\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.293689 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.293662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0eacb30d-a06a-468d-96f9-ff39199c9f76-klusterlet-config\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.298936 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.298914 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzzn\" (UniqueName: \"kubernetes.io/projected/0eacb30d-a06a-468d-96f9-ff39199c9f76-kube-api-access-fkzzn\") pod \"klusterlet-addon-workmgr-68f9646f9b-2w7n2\" (UID: \"0eacb30d-a06a-468d-96f9-ff39199c9f76\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.391506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391480 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48fz\" (UniqueName: \"kubernetes.io/projected/58e9f7ed-4d64-482a-9beb-418dd587dda7-kube-api-access-r48fz\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.391627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.391627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391566 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e9f7ed-4d64-482a-9beb-418dd587dda7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.391627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391582 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.391627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.391829 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.391633 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.392409 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.392363 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e9f7ed-4d64-482a-9beb-418dd587dda7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.394083 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.394061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-ca\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.394213 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.394194 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.394414 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.394394 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.394465 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.394451 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58e9f7ed-4d64-482a-9beb-418dd587dda7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.398998 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.398977 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48fz\" (UniqueName: \"kubernetes.io/projected/58e9f7ed-4d64-482a-9beb-418dd587dda7-kube-api-access-r48fz\") pod \"cluster-proxy-proxy-agent-59b56649c7-79bgm\" (UID: \"58e9f7ed-4d64-482a-9beb-418dd587dda7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.457057 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.457025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:36.497610 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.497576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:32:36.745700 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.745677 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2"] Sep 14 01:32:36.747979 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:32:36.747956 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eacb30d_a06a_468d_96f9_ff39199c9f76.slice/crio-875855f4bc154ed38aa4b953327d9a29aedb31149b0bcced44c5c3ec30845874 WatchSource:0}: Error finding container 875855f4bc154ed38aa4b953327d9a29aedb31149b0bcced44c5c3ec30845874: Status 404 returned error can't find the container with id 875855f4bc154ed38aa4b953327d9a29aedb31149b0bcced44c5c3ec30845874 Sep 14 01:32:36.792856 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:36.792835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm"] Sep 14 01:32:36.795480 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:32:36.795457 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58e9f7ed_4d64_482a_9beb_418dd587dda7.slice/crio-18ac732af22728de956d6a968b8be0300952e53213f1dde1c9d130c3115cf6c2 WatchSource:0}: Error finding container 18ac732af22728de956d6a968b8be0300952e53213f1dde1c9d130c3115cf6c2: Status 404 returned error can't find the container with id 18ac732af22728de956d6a968b8be0300952e53213f1dde1c9d130c3115cf6c2 Sep 14 01:32:37.329658 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:37.329618 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" event={"ID":"0eacb30d-a06a-468d-96f9-ff39199c9f76","Type":"ContainerStarted","Data":"875855f4bc154ed38aa4b953327d9a29aedb31149b0bcced44c5c3ec30845874"} Sep 14 01:32:37.330777 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:37.330757 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerStarted","Data":"18ac732af22728de956d6a968b8be0300952e53213f1dde1c9d130c3115cf6c2"} Sep 14 01:32:43.351880 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.351844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerStarted","Data":"0bc0478d95aaa6942c23d82f20cc3ef1fbdfe53a271f3683c46249456a19d9a5"} Sep 14 01:32:43.352358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.351889 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerStarted","Data":"19a6134ed89cad81dfd0af3ca341cad9f36c200784627784c2e26456ed43dd89"} Sep 14 01:32:43.352358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.351909 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerStarted","Data":"282c7e2c0f8d045a9b0d7ff4485163750ca8d10667c32432d2284af6d60219ff"} Sep 14 01:32:43.353265 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.353218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" event={"ID":"0eacb30d-a06a-468d-96f9-ff39199c9f76","Type":"ContainerStarted","Data":"91b58d06a16a134cbf878871899df0c62b9a6a96bdeb8c241f33d52b57ceff6e"} Sep 14 01:32:43.353382 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.353309 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:43.355116 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.355095 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:32:43.369317 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.369277 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" podStartSLOduration=1.603198396 podStartE2EDuration="7.36924476s" totalImagesPullingTime="5.766046364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:43.367575688 +0000 UTC m=+110.040691877" watchObservedRunningTime="2026-09-14 01:32:43.36924476 +0000 UTC m=+110.042360934" Sep 14 01:32:43.381585 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:32:43.381544 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" podStartSLOduration=1.566317249 podStartE2EDuration="7.381530159s" totalImagesPullingTime="5.81521291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:32:43.381014359 +0000 UTC m=+110.054130535" watchObservedRunningTime="2026-09-14 01:32:43.381530159 +0000 UTC m=+110.054646379" Sep 14 01:33:03.682223 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:33:03.682169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:33:03.682725 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:03.682347 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:33:03.682725 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:03.682429 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:05.68240967 +0000 UTC m=+252.355525824 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : secret "metrics-daemon-secret" not found Sep 14 01:33:31.677633 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:31.677591 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vwkwr" podUID="bf444def-59a8-44d1-8ac4-8240cdec1922" Sep 14 01:33:31.865603 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:31.865564 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:33:32.491190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:33:32.491161 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:33:36.609716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:33:36.609681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:33:36.610206 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:36.609825 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 01:33:36.610206 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:33:36.609889 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls podName:bf444def-59a8-44d1-8ac4-8240cdec1922 nodeName:}" failed. No retries permitted until 2026-09-14 01:35:38.609873967 +0000 UTC m=+285.282990124 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls") pod "dns-default-vwkwr" (UID: "bf444def-59a8-44d1-8ac4-8240cdec1922") : secret "dns-default-metrics-tls" not found Sep 14 01:33:42.855919 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:33:42.855892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:34:42.685093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:42.685060 2573 generic.go:356] "Generic (PLEG): container finished" podID="0eacb30d-a06a-468d-96f9-ff39199c9f76" containerID="91b58d06a16a134cbf878871899df0c62b9a6a96bdeb8c241f33d52b57ceff6e" exitCode=1 Sep 14 01:34:42.685437 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:42.685114 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" event={"ID":"0eacb30d-a06a-468d-96f9-ff39199c9f76","Type":"ContainerDied","Data":"91b58d06a16a134cbf878871899df0c62b9a6a96bdeb8c241f33d52b57ceff6e"} Sep 14 01:34:42.685437 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:42.685429 2573 scope.go:117] "RemoveContainer" containerID="91b58d06a16a134cbf878871899df0c62b9a6a96bdeb8c241f33d52b57ceff6e" Sep 14 01:34:43.354376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:43.354329 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:34:43.692801 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:43.692760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" event={"ID":"0eacb30d-a06a-468d-96f9-ff39199c9f76","Type":"ContainerStarted","Data":"1c5d095248e643a860d1e8171bc5498a377f5b8266abfac988f406b2ea90afed"} Sep 14 01:34:43.693151 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:43.692964 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:34:43.693552 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:34:43.693539 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68f9646f9b-2w7n2" Sep 14 01:35:05.707505 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:05.707468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:35:05.707949 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:05.707595 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 01:35:05.707949 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:05.707656 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs podName:8cc55a86-c227-45a3-93ba-721ba72f5346 nodeName:}" failed. No retries permitted until 2026-09-14 01:37:07.707642044 +0000 UTC m=+374.380758197 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs") pod "network-metrics-daemon-zflq9" (UID: "8cc55a86-c227-45a3-93ba-721ba72f5346") : secret "metrics-daemon-secret" not found Sep 14 01:35:08.442058 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.442022 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xzl9s"] Sep 14 01:35:08.446589 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.446565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.449226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.449203 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 01:35:08.449389 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.449243 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 01:35:08.450338 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.450306 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 01:35:08.450338 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.450322 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dqk8k\"" Sep 14 01:35:08.450338 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.450327 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 01:35:08.458560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.458538 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xzl9s"] Sep 14 01:35:08.526453 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.526428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbd6s\" (UniqueName: \"kubernetes.io/projected/9f4cd921-9291-452b-87ba-1e333d57050b-kube-api-access-jbd6s\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.526588 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.526469 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4cd921-9291-452b-87ba-1e333d57050b-crio-socket\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.526588 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.526506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4cd921-9291-452b-87ba-1e333d57050b-data-volume\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.526588 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.526557 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.526588 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.526583 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbd6s\" (UniqueName: \"kubernetes.io/projected/9f4cd921-9291-452b-87ba-1e333d57050b-kube-api-access-jbd6s\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627539 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4cd921-9291-452b-87ba-1e333d57050b-crio-socket\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627596 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4cd921-9291-452b-87ba-1e333d57050b-data-volume\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627668 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:08.627621 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:08.627917 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627690 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f4cd921-9291-452b-87ba-1e333d57050b-crio-socket\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.627917 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:08.627723 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:09.127700967 +0000 UTC m=+255.800817121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : configmap "kube-rbac-proxy" not found Sep 14 01:35:08.627917 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:08.627766 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:08.627917 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:08.627841 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:09.127823874 +0000 UTC m=+255.800940029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:08.628064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.627916 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f4cd921-9291-452b-87ba-1e333d57050b-data-volume\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:08.635786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:08.635759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbd6s\" (UniqueName: \"kubernetes.io/projected/9f4cd921-9291-452b-87ba-1e333d57050b-kube-api-access-jbd6s\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:09.131454 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:09.131415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:09.131454 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:09.131460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:09.131685 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:09.131574 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:09.131685 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:09.131578 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:09.131685 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:09.131649 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:10.13163429 +0000 UTC m=+256.804750460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : configmap "kube-rbac-proxy" not found Sep 14 01:35:09.131685 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:09.131663 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:10.13165736 +0000 UTC m=+256.804773512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:10.138373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:10.138336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:10.138373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:10.138373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:10.138853 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:10.138466 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:10.138853 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:10.138516 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:10.138853 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:10.138543 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.138521752 +0000 UTC m=+258.811637905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : configmap "kube-rbac-proxy" not found Sep 14 01:35:10.138853 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:10.138590 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:12.138573751 +0000 UTC m=+258.811689922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:10.165607 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:10.165575 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pflr_5206f48c-891c-4b88-b4b2-38c4cbc06a0b/dns-node-resolver/0.log" Sep 14 01:35:11.366170 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:11.366148 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kbv7x_00f4c9be-6791-4072-adfe-37b3b288e7e3/node-ca/0.log" Sep 14 01:35:12.154357 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:12.154320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:12.154357 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:12.154358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:12.154547 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:12.154478 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:12.154547 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:12.154494 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:12.154547 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:12.154532 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:16.154516034 +0000 UTC m=+262.827632192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : configmap "kube-rbac-proxy" not found Sep 14 01:35:12.154653 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:12.154554 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:16.154537787 +0000 UTC m=+262.827653947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:12.367007 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:12.366976 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nk9kd_36788e14-cf89-4938-8495-fafa3349508a/serve-healthcheck-canary/0.log" Sep 14 01:35:16.185328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:16.185222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:16.185328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:16.185293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:16.185766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:16.185366 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 01:35:16.185766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:16.185398 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 01:35:16.185766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:16.185421 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:24.185407777 +0000 UTC m=+270.858523930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : configmap "kube-rbac-proxy" not found Sep 14 01:35:16.185766 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:16.185471 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls podName:9f4cd921-9291-452b-87ba-1e333d57050b nodeName:}" failed. No retries permitted until 2026-09-14 01:35:24.18545118 +0000 UTC m=+270.858567347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzl9s" (UID: "9f4cd921-9291-452b-87ba-1e333d57050b") : secret "insights-runtime-extractor-tls" not found Sep 14 01:35:24.241986 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.241946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:24.241986 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.241989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:24.242536 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.242518 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f4cd921-9291-452b-87ba-1e333d57050b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:24.244389 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.244366 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f4cd921-9291-452b-87ba-1e333d57050b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzl9s\" (UID: \"9f4cd921-9291-452b-87ba-1e333d57050b\") " pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:24.354898 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.354872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xzl9s" Sep 14 01:35:24.624636 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.624560 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xzl9s"] Sep 14 01:35:24.628050 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:24.628027 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4cd921_9291_452b_87ba_1e333d57050b.slice/crio-c2e55557b5ac42b5cc617a5a35e274b1355704bfab3ee683f2afbc647de334c8 WatchSource:0}: Error finding container c2e55557b5ac42b5cc617a5a35e274b1355704bfab3ee683f2afbc647de334c8: Status 404 returned error can't find the container with id c2e55557b5ac42b5cc617a5a35e274b1355704bfab3ee683f2afbc647de334c8 Sep 14 01:35:24.816407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.816371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzl9s" event={"ID":"9f4cd921-9291-452b-87ba-1e333d57050b","Type":"ContainerStarted","Data":"d35f4713e865b1b35f312b260c54adb35407a5ed505e6f8efd964f14ffe5cd85"} Sep 14 01:35:24.816563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:24.816431 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzl9s" event={"ID":"9f4cd921-9291-452b-87ba-1e333d57050b","Type":"ContainerStarted","Data":"c2e55557b5ac42b5cc617a5a35e274b1355704bfab3ee683f2afbc647de334c8"} Sep 14 01:35:25.822760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:25.822716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzl9s" event={"ID":"9f4cd921-9291-452b-87ba-1e333d57050b","Type":"ContainerStarted","Data":"6340ee17ce93090a299ac5235b12a9551b492e5372a282f8c18f52dfd9f191a2"} Sep 14 01:35:26.829341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:26.827780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzl9s" event={"ID":"9f4cd921-9291-452b-87ba-1e333d57050b","Type":"ContainerStarted","Data":"e8c16564f029264144c3ce330db88814cefbe0d99a965d65f0bc329194a29992"} Sep 14 01:35:26.846036 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:26.845987 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xzl9s" podStartSLOduration=17.023496553 podStartE2EDuration="18.845973265s" totalImagesPullingTime="1.822476712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 01:35:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:26.844417097 +0000 UTC m=+273.517533271" watchObservedRunningTime="2026-09-14 01:35:26.845973265 +0000 UTC m=+273.519089440" Sep 14 01:35:30.096666 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.096629 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-lf689"] Sep 14 01:35:30.099503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.099486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:30.101996 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.101973 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b92w8\"" Sep 14 01:35:30.101996 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.101991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 01:35:30.102170 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.101991 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 01:35:30.107616 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.107596 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lf689"] Sep 14 01:35:30.185123 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.185089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbld\" (UniqueName: \"kubernetes.io/projected/45f2ced9-74a7-4414-af03-eeb98cdd90d7-kube-api-access-2vbld\") pod \"downloads-76f99584f4-lf689\" (UID: \"45f2ced9-74a7-4414-af03-eeb98cdd90d7\") " pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:30.285631 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.285605 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbld\" (UniqueName: \"kubernetes.io/projected/45f2ced9-74a7-4414-af03-eeb98cdd90d7-kube-api-access-2vbld\") pod \"downloads-76f99584f4-lf689\" (UID: \"45f2ced9-74a7-4414-af03-eeb98cdd90d7\") " pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:30.293301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.293281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbld\" (UniqueName: \"kubernetes.io/projected/45f2ced9-74a7-4414-af03-eeb98cdd90d7-kube-api-access-2vbld\") pod \"downloads-76f99584f4-lf689\" (UID: \"45f2ced9-74a7-4414-af03-eeb98cdd90d7\") " pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:30.409501 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.409464 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:30.676163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.676075 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-lf689"] Sep 14 01:35:30.678591 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:30.678564 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f2ced9_74a7_4414_af03_eeb98cdd90d7.slice/crio-4b0813ab15bfefc10fd4281c8a992729c44234e3c46b6b85aa0975dc2344bde0 WatchSource:0}: Error finding container 4b0813ab15bfefc10fd4281c8a992729c44234e3c46b6b85aa0975dc2344bde0: Status 404 returned error can't find the container with id 4b0813ab15bfefc10fd4281c8a992729c44234e3c46b6b85aa0975dc2344bde0 Sep 14 01:35:30.840671 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:30.840637 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lf689" event={"ID":"45f2ced9-74a7-4414-af03-eeb98cdd90d7","Type":"ContainerStarted","Data":"4b0813ab15bfefc10fd4281c8a992729c44234e3c46b6b85aa0975dc2344bde0"} Sep 14 01:35:35.492288 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:35.492218 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vwkwr" podUID="bf444def-59a8-44d1-8ac4-8240cdec1922" Sep 14 01:35:35.855863 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:35.855783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:35:36.498942 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:36.498884 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" podUID="58e9f7ed-4d64-482a-9beb-418dd587dda7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 01:35:37.338816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.338781 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t6659"] Sep 14 01:35:37.342014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.341988 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.344363 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.344340 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 01:35:37.344496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.344422 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 01:35:37.344496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.344429 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 01:35:37.345394 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.345179 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wvnzr\"" Sep 14 01:35:37.345394 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.345237 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 01:35:37.345394 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.345277 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 01:35:37.345610 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.345515 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 01:35:37.438943 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.438905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.438943 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.438945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-textfile\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-wtmp\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439080 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-metrics-client-ca\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-sys\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q88km\" (UniqueName: \"kubernetes.io/projected/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-kube-api-access-q88km\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439196 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-root\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.439339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.439214 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540475 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88km\" (UniqueName: \"kubernetes.io/projected/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-kube-api-access-q88km\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-root\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:37.540660 2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540696 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-root\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540728 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:37.540765 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls podName:8ce5fdda-b5f1-4193-9f6d-3ec443f4378d nodeName:}" failed. No retries permitted until 2026-09-14 01:35:38.040746748 +0000 UTC m=+284.713862904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls") pod "node-exporter-t6659" (UID: "8ce5fdda-b5f1-4193-9f6d-3ec443f4378d") : secret "node-exporter-tls" not found Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-textfile\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.540900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-wtmp\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540993 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-metrics-client-ca\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.541024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-sys\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.541115 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-sys\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.540994 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-wtmp\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.541130 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-textfile\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.541352 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-accelerators-collector-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.541668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.541465 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-metrics-client-ca\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.543508 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.543487 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:37.548486 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:37.548467 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88km\" (UniqueName: \"kubernetes.io/projected/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-kube-api-access-q88km\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:38.044526 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.044497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:38.047386 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.047359 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ce5fdda-b5f1-4193-9f6d-3ec443f4378d-node-exporter-tls\") pod \"node-exporter-t6659\" (UID: \"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d\") " pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:38.254002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.253964 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t6659" Sep 14 01:35:38.266724 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:38.266696 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce5fdda_b5f1_4193_9f6d_3ec443f4378d.slice/crio-0d902f2a8d9bcfc9cc7d0914512d17a3e9d8a5e37eb7f98f81f2766cfce604fa WatchSource:0}: Error finding container 0d902f2a8d9bcfc9cc7d0914512d17a3e9d8a5e37eb7f98f81f2766cfce604fa: Status 404 returned error can't find the container with id 0d902f2a8d9bcfc9cc7d0914512d17a3e9d8a5e37eb7f98f81f2766cfce604fa Sep 14 01:35:38.648752 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.648708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:35:38.651789 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.651753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf444def-59a8-44d1-8ac4-8240cdec1922-metrics-tls\") pod \"dns-default-vwkwr\" (UID: \"bf444def-59a8-44d1-8ac4-8240cdec1922\") " pod="openshift-dns/dns-default-vwkwr" Sep 14 01:35:38.859518 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.859397 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jfmw6\"" Sep 14 01:35:38.867515 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.867246 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:35:38.872593 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:38.869778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t6659" event={"ID":"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d","Type":"ContainerStarted","Data":"0d902f2a8d9bcfc9cc7d0914512d17a3e9d8a5e37eb7f98f81f2766cfce604fa"} Sep 14 01:35:39.157290 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:39.157240 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vwkwr"] Sep 14 01:35:39.300476 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:39.300399 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf444def_59a8_44d1_8ac4_8240cdec1922.slice/crio-5797bc29ddc3629b425db3b049ea550971158c5f1a1bd508894eb9aeae45a444 WatchSource:0}: Error finding container 5797bc29ddc3629b425db3b049ea550971158c5f1a1bd508894eb9aeae45a444: Status 404 returned error can't find the container with id 5797bc29ddc3629b425db3b049ea550971158c5f1a1bd508894eb9aeae45a444 Sep 14 01:35:39.874975 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:39.874934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t6659" event={"ID":"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d","Type":"ContainerStarted","Data":"d494ae77226ad3c67aa1408e8ba7ee48527bf629465ee17a691db8b424e6e110"} Sep 14 01:35:39.876491 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:39.876456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwkwr" event={"ID":"bf444def-59a8-44d1-8ac4-8240cdec1922","Type":"ContainerStarted","Data":"5797bc29ddc3629b425db3b049ea550971158c5f1a1bd508894eb9aeae45a444"} Sep 14 01:35:41.592818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.592781 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dd679c59-2zsph"] Sep 14 01:35:41.597231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.597204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.599421 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.599395 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 01:35:41.599566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.599454 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hmnp8\"" Sep 14 01:35:41.599566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.599460 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 01:35:41.600442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.600422 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 01:35:41.600554 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.600422 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 01:35:41.600554 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.600488 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2i81c052q55jv\"" Sep 14 01:35:41.604598 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.604578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd679c59-2zsph"] Sep 14 01:35:41.777691 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-metrics-server-audit-profiles\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777691 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777694 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-audit-log\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-client-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8m8\" (UniqueName: \"kubernetes.io/projected/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-kube-api-access-sm8m8\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777785 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-tls\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-client-certs\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.777950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.777889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.878799 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.878713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-client-certs\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.878799 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.878757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879017 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.878936 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-metrics-server-audit-profiles\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879017 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.878977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-audit-log\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879156 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.879116 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-client-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879242 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.879160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8m8\" (UniqueName: \"kubernetes.io/projected/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-kube-api-access-sm8m8\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879242 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.879208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-tls\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879536 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.879499 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-audit-log\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.879602 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.879542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.880380 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.880356 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-metrics-server-audit-profiles\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.881916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.881881 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-client-certs\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.882223 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.882196 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-client-ca-bundle\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.882332 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.882310 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-secret-metrics-server-tls\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.886225 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.886197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8m8\" (UniqueName: \"kubernetes.io/projected/eda5e9cb-3fc4-4ae0-a77c-f8882119733f-kube-api-access-sm8m8\") pod \"metrics-server-5dd679c59-2zsph\" (UID: \"eda5e9cb-3fc4-4ae0-a77c-f8882119733f\") " pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:41.908100 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:41.908080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:35:42.505644 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.505613 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f6759447f-5mwn9"] Sep 14 01:35:42.508852 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.508833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.511483 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.511386 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 14 01:35:42.511483 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.511409 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 14 01:35:42.512005 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.511955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2fv7q\"" Sep 14 01:35:42.512005 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.511979 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 14 01:35:42.512171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.511955 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 14 01:35:42.512471 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.512450 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 14 01:35:42.517897 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.517786 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 14 01:35:42.521377 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.521359 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6759447f-5mwn9"] Sep 14 01:35:42.685346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-federate-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685457 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-metrics-client-ca\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-serving-certs-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.685767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.685640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bck86\" (UniqueName: \"kubernetes.io/projected/a840611c-42ba-4c5a-b35e-529aa9c28682-kube-api-access-bck86\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786692 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786600 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-federate-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786692 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786692 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786954 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-metrics-client-ca\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786954 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786954 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-serving-certs-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.786954 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.786946 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bck86\" (UniqueName: \"kubernetes.io/projected/a840611c-42ba-4c5a-b35e-529aa9c28682-kube-api-access-bck86\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.787166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.787002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.787836 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.787803 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-metrics-client-ca\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.787974 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.787956 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.788137 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.788112 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a840611c-42ba-4c5a-b35e-529aa9c28682-serving-certs-ca-bundle\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.789857 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.789836 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.790070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.790047 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-secret-telemeter-client\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.790172 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.790051 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-federate-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.790333 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.790296 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a840611c-42ba-4c5a-b35e-529aa9c28682-telemeter-client-tls\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.794948 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.794915 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bck86\" (UniqueName: \"kubernetes.io/projected/a840611c-42ba-4c5a-b35e-529aa9c28682-kube-api-access-bck86\") pod \"telemeter-client-7f6759447f-5mwn9\" (UID: \"a840611c-42ba-4c5a-b35e-529aa9c28682\") " pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:42.821197 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:42.821073 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" Sep 14 01:35:43.476022 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.475985 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:35:43.479740 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.479709 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.482510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.482480 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 01:35:43.482827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.482802 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 01:35:43.482942 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.482849 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 01:35:43.482942 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.482902 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aibv9pnvt0b41\"" Sep 14 01:35:43.483985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.483960 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 01:35:43.484092 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484009 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 01:35:43.484092 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484070 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pjtnt\"" Sep 14 01:35:43.484467 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484447 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484651 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484664 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484714 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484762 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484792 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 01:35:43.484934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.484835 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 01:35:43.487872 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.487848 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 01:35:43.490686 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.490668 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 01:35:43.498424 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.498404 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:35:43.594815 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.594815 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.594986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595279 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595307 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595355 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpdp\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595441 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.595874 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.595464 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.696423 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.696423 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696427 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.696920 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696447 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.696920 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696481 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.696920 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.696960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697112 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697193 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697275 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697867 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpdp\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697867 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.697974 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.697939 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.699864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.699023 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.699864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.699671 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.700050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.699874 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.700453 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.700403 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.700687 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.700540 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.700825 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.700802 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.701426 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.701166 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.701426 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.701312 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.702159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.702096 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.702629 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.702578 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.702728 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.702662 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.703646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.703598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.704026 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.703986 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.704117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.704079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.704727 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.704704 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.704951 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.704930 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.709564 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.709545 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpdp\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp\") pod \"prometheus-k8s-0\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:43.791734 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:43.791659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:35:45.857145 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:35:45.857101 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zflq9" podUID="8cc55a86-c227-45a3-93ba-721ba72f5346" Sep 14 01:35:46.500165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.499699 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" podUID="58e9f7ed-4d64-482a-9beb-418dd587dda7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 01:35:46.698352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.698322 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dd679c59-2zsph"] Sep 14 01:35:46.701157 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:46.701125 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda5e9cb_3fc4_4ae0_a77c_f8882119733f.slice/crio-346e8a9418db927854315c77f87627b4e5df19697540f57110f15b2bf338553c WatchSource:0}: Error finding container 346e8a9418db927854315c77f87627b4e5df19697540f57110f15b2bf338553c: Status 404 returned error can't find the container with id 346e8a9418db927854315c77f87627b4e5df19697540f57110f15b2bf338553c Sep 14 01:35:46.708858 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:46.708825 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda840611c_42ba_4c5a_b35e_529aa9c28682.slice/crio-14adba5f6ebc005f9698ae62657a94b268d0caae73ab527925489a9d513724b4 WatchSource:0}: Error finding container 14adba5f6ebc005f9698ae62657a94b268d0caae73ab527925489a9d513724b4: Status 404 returned error can't find the container with id 14adba5f6ebc005f9698ae62657a94b268d0caae73ab527925489a9d513724b4 Sep 14 01:35:46.709134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.709058 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6759447f-5mwn9"] Sep 14 01:35:46.710849 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.710786 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:35:46.712519 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:35:46.712495 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df7ab26_6f1f_44f5_9925_ff8ccd29d328.slice/crio-4d381a597f41a00cb9af7ec2dda0951f5d7f6eeecbe8db0b0b44a56e68f21c66 WatchSource:0}: Error finding container 4d381a597f41a00cb9af7ec2dda0951f5d7f6eeecbe8db0b0b44a56e68f21c66: Status 404 returned error can't find the container with id 4d381a597f41a00cb9af7ec2dda0951f5d7f6eeecbe8db0b0b44a56e68f21c66 Sep 14 01:35:46.905977 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.905500 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-lf689" event={"ID":"45f2ced9-74a7-4414-af03-eeb98cdd90d7","Type":"ContainerStarted","Data":"05b42452a648041904fc9126f58b89b791e7d0337a2503beb9abc4c2e5c88ede"} Sep 14 01:35:46.905977 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.905809 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:46.907809 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.907780 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" event={"ID":"eda5e9cb-3fc4-4ae0-a77c-f8882119733f","Type":"ContainerStarted","Data":"346e8a9418db927854315c77f87627b4e5df19697540f57110f15b2bf338553c"} Sep 14 01:35:46.909753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.909731 2573 generic.go:356] "Generic (PLEG): container finished" podID="8ce5fdda-b5f1-4193-9f6d-3ec443f4378d" containerID="d494ae77226ad3c67aa1408e8ba7ee48527bf629465ee17a691db8b424e6e110" exitCode=0 Sep 14 01:35:46.909869 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.909803 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t6659" event={"ID":"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d","Type":"ContainerDied","Data":"d494ae77226ad3c67aa1408e8ba7ee48527bf629465ee17a691db8b424e6e110"} Sep 14 01:35:46.911271 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.911227 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" event={"ID":"a840611c-42ba-4c5a-b35e-529aa9c28682","Type":"ContainerStarted","Data":"14adba5f6ebc005f9698ae62657a94b268d0caae73ab527925489a9d513724b4"} Sep 14 01:35:46.913386 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.913351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"4d381a597f41a00cb9af7ec2dda0951f5d7f6eeecbe8db0b0b44a56e68f21c66"} Sep 14 01:35:46.915159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.915142 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-lf689" Sep 14 01:35:46.920924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:46.920880 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-lf689" podStartSLOduration=1.0719307279999999 podStartE2EDuration="16.920861615s" totalImagesPullingTime="15.848930887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:46.919370133 +0000 UTC m=+293.592486301" watchObservedRunningTime="2026-09-14 01:35:46.920861615 +0000 UTC m=+293.593977791" Sep 14 01:35:47.923808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:47.923266 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t6659" event={"ID":"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d","Type":"ContainerStarted","Data":"55c329121b6dcd598c5d7562c8b2089906da7858f94df68c82726324c17feee7"} Sep 14 01:35:47.923808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:47.923335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t6659" event={"ID":"8ce5fdda-b5f1-4193-9f6d-3ec443f4378d","Type":"ContainerStarted","Data":"f079749b4f62323c31f94136f19fd953b5fe54a1b23bbb2b62aada68251858fd"} Sep 14 01:35:47.945513 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:47.945453 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t6659" podStartSLOduration=9.791825341 podStartE2EDuration="10.945433564s" totalImagesPullingTime="1.077147444s" totalInitContainerRuntime="76.460779ms" isStatefulPod=true podCreationTimestamp="2026-09-14 01:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:47.944386754 +0000 UTC m=+294.617502930" watchObservedRunningTime="2026-09-14 01:35:47.945433564 +0000 UTC m=+294.618549740" Sep 14 01:35:51.942855 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.942810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" event={"ID":"a840611c-42ba-4c5a-b35e-529aa9c28682","Type":"ContainerStarted","Data":"4bd3a42cb40103b8466239a275c35f24077465546f49ac78bdd34035ac680420"} Sep 14 01:35:51.943354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.942885 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" event={"ID":"a840611c-42ba-4c5a-b35e-529aa9c28682","Type":"ContainerStarted","Data":"9ae302194e1403b3359c1b786c75273b1d369386cdc1c75bfec49c38a95acdbb"} Sep 14 01:35:51.943354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.942914 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" event={"ID":"a840611c-42ba-4c5a-b35e-529aa9c28682","Type":"ContainerStarted","Data":"fa62193a0fa3688b27a3e267a98a87a038df52ce396678b7cd940db27e64a9da"} Sep 14 01:35:51.944851 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.944818 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" exitCode=0 Sep 14 01:35:51.944959 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.944908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} Sep 14 01:35:51.946881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.946853 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" event={"ID":"eda5e9cb-3fc4-4ae0-a77c-f8882119733f","Type":"ContainerStarted","Data":"f3918ff7758c91ab1c79d5fe8942677470527724fdfbf3de15d264cfe25a8256"} Sep 14 01:35:51.949385 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.949360 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwkwr" event={"ID":"bf444def-59a8-44d1-8ac4-8240cdec1922","Type":"ContainerStarted","Data":"e0c5ee26b23f991f5412ecc27d782c85787ca9c4668bca08a748a0468d002c9b"} Sep 14 01:35:51.949497 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.949397 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vwkwr" event={"ID":"bf444def-59a8-44d1-8ac4-8240cdec1922","Type":"ContainerStarted","Data":"dd4081970442ee901eb10a7fcd4a3e84430b2520e07ca2045be88ebd1b662357"} Sep 14 01:35:51.949497 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.949435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:35:51.966831 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:51.966761 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f6759447f-5mwn9" podStartSLOduration=5.7864216729999995 podStartE2EDuration="9.966745245s" totalImagesPullingTime="4.180323572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:51.966650541 +0000 UTC m=+298.639766719" watchObservedRunningTime="2026-09-14 01:35:51.966745245 +0000 UTC m=+298.639861421" Sep 14 01:35:52.013919 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:52.013870 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vwkwr" podStartSLOduration=252.430683435 podStartE2EDuration="4m24.013853031s" totalImagesPullingTime="11.583169596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:52.013144919 +0000 UTC m=+298.686261094" watchObservedRunningTime="2026-09-14 01:35:52.013853031 +0000 UTC m=+298.686969206" Sep 14 01:35:52.032035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:52.031988 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" podStartSLOduration=6.848904178 podStartE2EDuration="11.031967069s" totalImagesPullingTime="4.183062891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:52.030517292 +0000 UTC m=+298.703633468" watchObservedRunningTime="2026-09-14 01:35:52.031967069 +0000 UTC m=+298.705083245" Sep 14 01:35:53.803343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:53.803161 2573 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 01:35:56.333691 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.333524 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:35:56.498818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.498782 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" podUID="58e9f7ed-4d64-482a-9beb-418dd587dda7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 14 01:35:56.498974 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.498847 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" Sep 14 01:35:56.499370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.499340 2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"0bc0478d95aaa6942c23d82f20cc3ef1fbdfe53a271f3683c46249456a19d9a5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 14 01:35:56.499421 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.499391 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" podUID="58e9f7ed-4d64-482a-9beb-418dd587dda7" containerName="service-proxy" containerID="cri-o://0bc0478d95aaa6942c23d82f20cc3ef1fbdfe53a271f3683c46249456a19d9a5" gracePeriod=30 Sep 14 01:35:56.974084 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.973974 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} Sep 14 01:35:56.974343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.974101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} Sep 14 01:35:56.976900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.976876 2573 generic.go:356] "Generic (PLEG): container finished" podID="58e9f7ed-4d64-482a-9beb-418dd587dda7" containerID="0bc0478d95aaa6942c23d82f20cc3ef1fbdfe53a271f3683c46249456a19d9a5" exitCode=2 Sep 14 01:35:56.977020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.976933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerDied","Data":"0bc0478d95aaa6942c23d82f20cc3ef1fbdfe53a271f3683c46249456a19d9a5"} Sep 14 01:35:56.977020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:56.976971 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59b56649c7-79bgm" event={"ID":"58e9f7ed-4d64-482a-9beb-418dd587dda7","Type":"ContainerStarted","Data":"414136cef1d79e1bdd9e93dd0ffb6bfb8de283ccaf27aaae100f9d0796b2d88a"} Sep 14 01:35:58.988308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:58.988209 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} Sep 14 01:35:58.988308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:58.988300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} Sep 14 01:35:58.988786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:58.988327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} Sep 14 01:35:58.988786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:58.988351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerStarted","Data":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} Sep 14 01:35:59.015648 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:59.015602 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.604278215 podStartE2EDuration="16.015586706s" totalImagesPullingTime="10.392246062s" totalInitContainerRuntime="19.062429ms" isStatefulPod=false podCreationTimestamp="2026-09-14 01:35:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:35:59.01432877 +0000 UTC m=+305.687444945" watchObservedRunningTime="2026-09-14 01:35:59.015586706 +0000 UTC m=+305.688702872" Sep 14 01:35:59.855373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:35:59.855337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:36:01.909143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:01.909114 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:36:01.909637 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:01.909189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:36:01.957456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:01.957435 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vwkwr" Sep 14 01:36:03.792313 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:03.792263 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:36:21.913376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:21.913350 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:36:21.917267 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:21.917232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dd679c59-2zsph" Sep 14 01:36:43.792777 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:43.792740 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:36:43.811656 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:43.811631 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:36:44.166672 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:36:44.166650 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:01.843932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.843899 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844220 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="prometheus" containerID="cri-o://86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" gracePeriod=600 Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844270 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy" containerID="cri-o://5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" gracePeriod=600 Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844287 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-web" containerID="cri-o://54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" gracePeriod=600 Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844287 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="thanos-sidecar" containerID="cri-o://5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" gracePeriod=600 Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844321 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-thanos" containerID="cri-o://09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" gracePeriod=600 Sep 14 01:37:01.844510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:01.844295 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="config-reloader" containerID="cri-o://17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" gracePeriod=600 Sep 14 01:37:02.093754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.093730 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.190496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190470 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndpdp\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190503 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190537 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190563 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190591 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190634 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190660 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190707 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.190749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190747 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190779 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190807 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190848 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190876 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190903 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190935 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190968 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.190997 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191020 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191131 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191055 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls\") pod \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\" (UID: \"4df7ab26-6f1f-44f5-9925-ff8ccd29d328\") " Sep 14 01:37:02.191613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.191613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191323 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.191613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191371 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.191765 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.191730 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.193322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.193285 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.193448 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.193370 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.193523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.193396 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:37:02.194121 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.193847 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.194930 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.194747 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.194930 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.194846 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.195096 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.195036 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.195096 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.194693 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.195352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.195231 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.195508 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.195464 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp" (OuterVolumeSpecName: "kube-api-access-ndpdp") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "kube-api-access-ndpdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:37:02.196111 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196085 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.196111 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196094 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out" (OuterVolumeSpecName: "config-out") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:37:02.196361 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196330 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config" (OuterVolumeSpecName: "config") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.196361 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196345 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:37:02.196701 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196687 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:37:02.196971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.196949 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.204729 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.204711 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config" (OuterVolumeSpecName: "web-config") pod "4df7ab26-6f1f-44f5-9925-ff8ccd29d328" (UID: "4df7ab26-6f1f-44f5-9925-ff8ccd29d328"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:37:02.214964 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.214941 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.214980 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.214987 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.214993 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.214998 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215005 2573 generic.go:356] "Generic (PLEG): container finished" podID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" exitCode=0 Sep 14 01:37:02.215042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215018 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215146 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215198 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215211 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.215222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215223 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} Sep 14 01:37:02.215465 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} Sep 14 01:37:02.215465 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.215293 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4df7ab26-6f1f-44f5-9925-ff8ccd29d328","Type":"ContainerDied","Data":"4d381a597f41a00cb9af7ec2dda0951f5d7f6eeecbe8db0b0b44a56e68f21c66"} Sep 14 01:37:02.223373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.223354 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.230032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.230015 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.236596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.236577 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.238514 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.238491 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:02.242551 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.242528 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:02.244169 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.244151 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.250986 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.250972 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.258590 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.258574 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.264853 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.264835 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.265148 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.265129 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.265190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265156 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.265190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265172 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.265451 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.265433 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.265504 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265456 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.265504 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265472 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.265697 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.265675 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.265735 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265702 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.265735 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265717 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.265949 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.265935 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.266054 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265952 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.266054 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.265965 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.266213 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.266197 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.266309 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266217 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.266309 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266234 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.266600 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.266583 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.266656 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266603 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.266656 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266653 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.266992 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:37:02.266930 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.266992 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266950 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.266992 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.266964 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.267196 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267179 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.267235 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267196 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.267646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267624 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.267646 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267646 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.267959 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267942 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.267959 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.267958 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.268227 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268210 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.268301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268229 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.268408 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268390 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:02.268498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268483 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.268549 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268499 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.268642 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268627 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="thanos-sidecar" Sep 14 01:37:02.268642 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268642 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="thanos-sidecar" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268652 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-thanos" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268661 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-thanos" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268677 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="init-config-reloader" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268683 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="init-config-reloader" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268692 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="config-reloader" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268697 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="config-reloader" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268703 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268707 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268719 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="prometheus" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268724 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="prometheus" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268731 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-web" Sep 14 01:37:02.268754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268736 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-web" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268776 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="thanos-sidecar" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268783 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268790 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="config-reloader" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268720 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268809 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268796 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="prometheus" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268855 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-web" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.268863 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" containerName="kube-rbac-proxy-thanos" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269054 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.269143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269075 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.269577 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269320 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.269577 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269337 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.269672 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269579 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.269672 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269601 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.269840 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269819 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.269889 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.269842 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.270098 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270077 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.270150 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270100 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.270361 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270344 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.270422 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270361 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.270576 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270559 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.270615 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270577 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.270792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270776 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.270844 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.270793 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.271053 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271034 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.271097 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271055 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.271323 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271306 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.271383 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271323 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.271559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271542 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.271601 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271561 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.271792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271775 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.271842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271792 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.272002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.271980 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.272042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272003 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.272215 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272200 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.272269 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272216 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.272432 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272415 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.272476 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272432 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.272627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272610 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.272663 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272628 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.272862 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272846 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.272918 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.272862 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.273105 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273085 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.273157 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273106 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.273370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273353 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.273409 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273370 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.273566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273549 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.273626 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273565 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.273822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273801 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.273822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273822 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.273950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.273872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.274073 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274051 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.274165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274075 2573 scope.go:117] "RemoveContainer" containerID="09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2" Sep 14 01:37:02.274568 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274550 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2"} err="failed to get container status \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": rpc error: code = NotFound desc = could not find container \"09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2\": container with ID starting with 09e07e668e8d287f6356f14752b2b4f4120d8390cbba8e08e822a7abdebe08d2 not found: ID does not exist" Sep 14 01:37:02.274568 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274567 2573 scope.go:117] "RemoveContainer" containerID="5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d" Sep 14 01:37:02.274837 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274817 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d"} err="failed to get container status \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": rpc error: code = NotFound desc = could not find container \"5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d\": container with ID starting with 5fb632c21d34b37015f8bfe5680985bd0d011aedea72cd3835d84f47bbc3233d not found: ID does not exist" Sep 14 01:37:02.274887 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.274839 2573 scope.go:117] "RemoveContainer" containerID="54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b" Sep 14 01:37:02.275085 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275066 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b"} err="failed to get container status \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": rpc error: code = NotFound desc = could not find container \"54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b\": container with ID starting with 54e5390673c5b5e2bfcd1a0dd47add15fe072f2ea51a87b90d99deedf7d27b3b not found: ID does not exist" Sep 14 01:37:02.275085 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275084 2573 scope.go:117] "RemoveContainer" containerID="5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38" Sep 14 01:37:02.275370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275349 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38"} err="failed to get container status \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": rpc error: code = NotFound desc = could not find container \"5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38\": container with ID starting with 5986e40f04f8ae8fc4a0ebea7008415dd5ab03c95cd05f483cd481f6d6a0bd38 not found: ID does not exist" Sep 14 01:37:02.275370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275366 2573 scope.go:117] "RemoveContainer" containerID="17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051" Sep 14 01:37:02.275638 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275618 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051"} err="failed to get container status \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": rpc error: code = NotFound desc = could not find container \"17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051\": container with ID starting with 17b639a02b63c3b9558844db4b650750b97a3bbd36d38e427a7c76a4f996d051 not found: ID does not exist" Sep 14 01:37:02.275638 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275638 2573 scope.go:117] "RemoveContainer" containerID="86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d" Sep 14 01:37:02.275861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275840 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d"} err="failed to get container status \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": rpc error: code = NotFound desc = could not find container \"86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d\": container with ID starting with 86dd7b10483f421bd9f67a04c990b346479acefd1aaa86842f78b6783c45020d not found: ID does not exist" Sep 14 01:37:02.275909 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.275864 2573 scope.go:117] "RemoveContainer" containerID="4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca" Sep 14 01:37:02.276093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.276077 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca"} err="failed to get container status \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": rpc error: code = NotFound desc = could not find container \"4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca\": container with ID starting with 4dc91e7cfd0e08e930dfb60544d0c1e14cb50b590f941dae6c59a9ef21dd73ca not found: ID does not exist" Sep 14 01:37:02.276596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.276581 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 01:37:02.276848 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.276830 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 01:37:02.276967 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.276954 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 01:37:02.277032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.276967 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aibv9pnvt0b41\"" Sep 14 01:37:02.277328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277311 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 01:37:02.277413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277350 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 01:37:02.277738 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277679 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 01:37:02.277916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277751 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 01:37:02.277916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277785 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 01:37:02.277916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.277864 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 01:37:02.278096 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.278079 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 01:37:02.278152 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.278083 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 01:37:02.278299 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.278245 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 01:37:02.278553 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.278535 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pjtnt\"" Sep 14 01:37:02.280207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.280176 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 01:37:02.282647 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.282630 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 01:37:02.286472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.286450 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:02.292502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292478 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-db\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292497 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config-out\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292506 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-tls-assets\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292514 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-config\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292522 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292531 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292540 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292549 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292556 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292564 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-metrics-client-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292572 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292579 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndpdp\" (UniqueName: \"kubernetes.io/projected/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-kube-api-access-ndpdp\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292589 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-kube-rbac-proxy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292597 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-metrics-client-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292605 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.292612 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292613 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-grpc-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.293023 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292622 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.293023 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292630 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.293023 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.292638 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4df7ab26-6f1f-44f5-9925-ff8ccd29d328-web-config\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:37:02.393804 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.393804 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.393983 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcwl\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-kube-api-access-xvcwl\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-config-out\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394322 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-web-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.394778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.394442 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.495632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495543 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.495632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495583 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.495632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcwl\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-kube-api-access-xvcwl\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.495632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495847 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.495972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496068 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-config-out\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496145 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-web-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496407 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.496622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.497560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496570 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.497560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.496596 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.497560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.497197 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.497560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.497322 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.498420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.498108 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.498420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.498303 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.499119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.498804 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.499119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.498827 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.499641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.499329 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.499641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.499351 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.499934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.499908 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1148453c-91a7-4f34-b89b-736843cff462-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.500230 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.500159 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.500995 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.500973 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.501079 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.501026 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.501298 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.501281 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-web-config\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.501730 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.501705 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1148453c-91a7-4f34-b89b-736843cff462-config-out\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.501794 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.501755 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.502019 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.502000 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1148453c-91a7-4f34-b89b-736843cff462-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.507238 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.507218 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcwl\" (UniqueName: \"kubernetes.io/projected/1148453c-91a7-4f34-b89b-736843cff462-kube-api-access-xvcwl\") pod \"prometheus-k8s-0\" (UID: \"1148453c-91a7-4f34-b89b-736843cff462\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.583554 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.583520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:02.862163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:02.861729 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 01:37:02.864775 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:37:02.864747 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1148453c_91a7_4f34_b89b_736843cff462.slice/crio-0dff52a674eebd744819e116782e6345140564b3185692108ff565621d8f3f52 WatchSource:0}: Error finding container 0dff52a674eebd744819e116782e6345140564b3185692108ff565621d8f3f52: Status 404 returned error can't find the container with id 0dff52a674eebd744819e116782e6345140564b3185692108ff565621d8f3f52 Sep 14 01:37:03.221469 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:03.221432 2573 generic.go:356] "Generic (PLEG): container finished" podID="1148453c-91a7-4f34-b89b-736843cff462" containerID="8b491549a2809febcb4eea3416500388f335c99019c2ef3bb09ecf2a39ed02a1" exitCode=0 Sep 14 01:37:03.221469 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:03.221471 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerDied","Data":"8b491549a2809febcb4eea3416500388f335c99019c2ef3bb09ecf2a39ed02a1"} Sep 14 01:37:03.221725 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:03.221524 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"0dff52a674eebd744819e116782e6345140564b3185692108ff565621d8f3f52"} Sep 14 01:37:03.861567 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:03.861487 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4df7ab26-6f1f-44f5-9925-ff8ccd29d328" path="/var/lib/kubelet/pods/4df7ab26-6f1f-44f5-9925-ff8ccd29d328/volumes" Sep 14 01:37:04.228225 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"b5cf28e883bf864331f52bb4caeed8208c5f80de75417f655c0fa7250d214c48"} Sep 14 01:37:04.228584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"caf01229f3fafe0df65dd5bcf06112daae86a9f408477890bbf0291124a40af6"} Sep 14 01:37:04.228584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"39664f7c5ed1cd232d98d8cd4e474ffac3625ef19dac908581de914890528a4a"} Sep 14 01:37:04.228584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228324 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"50e273b871e7e0ebe3ef6baa7f50b9bdb206f061238ec905e57efea3858cda88"} Sep 14 01:37:04.228584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"8d8b9b0c0133c95581813565e0503a478df3469f9b8d605ee61e423d44f4dc5e"} Sep 14 01:37:04.228584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.228371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1148453c-91a7-4f34-b89b-736843cff462","Type":"ContainerStarted","Data":"7f96facace1ecc6bee94b3831154058297a8dda523a6e00a2d80c123f16ee8db"} Sep 14 01:37:04.255610 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:04.254714 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.231157707 podStartE2EDuration="2.254694005s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.536298ms" isStatefulPod=false podCreationTimestamp="2026-09-14 01:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:37:04.252420539 +0000 UTC m=+370.925536714" watchObservedRunningTime="2026-09-14 01:37:04.254694005 +0000 UTC m=+370.927810179" Sep 14 01:37:07.584091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:07.584057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:37:07.743070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:07.743018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:37:07.746034 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:07.746008 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cc55a86-c227-45a3-93ba-721ba72f5346-metrics-certs\") pod \"network-metrics-daemon-zflq9\" (UID: \"8cc55a86-c227-45a3-93ba-721ba72f5346\") " pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:37:07.958437 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:07.958411 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gv4zq\"" Sep 14 01:37:07.966340 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:07.966318 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zflq9" Sep 14 01:37:08.236294 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:08.236247 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zflq9"] Sep 14 01:37:08.238886 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:37:08.238861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc55a86_c227_45a3_93ba_721ba72f5346.slice/crio-a49494df62e1d46cf2153c900072f93dafe622503aaed76a31a7e8a5477e58be WatchSource:0}: Error finding container a49494df62e1d46cf2153c900072f93dafe622503aaed76a31a7e8a5477e58be: Status 404 returned error can't find the container with id a49494df62e1d46cf2153c900072f93dafe622503aaed76a31a7e8a5477e58be Sep 14 01:37:08.245026 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:08.245002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zflq9" event={"ID":"8cc55a86-c227-45a3-93ba-721ba72f5346","Type":"ContainerStarted","Data":"a49494df62e1d46cf2153c900072f93dafe622503aaed76a31a7e8a5477e58be"} Sep 14 01:37:10.252983 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:10.252951 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zflq9" event={"ID":"8cc55a86-c227-45a3-93ba-721ba72f5346","Type":"ContainerStarted","Data":"114c1a148bc3e35fbdb167ddae8278c0fe816852db14df732595cd382a6171b9"} Sep 14 01:37:10.253369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:10.253001 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zflq9" event={"ID":"8cc55a86-c227-45a3-93ba-721ba72f5346","Type":"ContainerStarted","Data":"6e0ff638608b9894556ef79459bfb8f15c1a25ab16102428551de38c602aa865"} Sep 14 01:37:10.268743 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:37:10.268685 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zflq9" podStartSLOduration=375.273219564 podStartE2EDuration="6m16.268667014s" totalImagesPullingTime="995.44745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:37:10.267213791 +0000 UTC m=+376.940329967" watchObservedRunningTime="2026-09-14 01:37:10.268667014 +0000 UTC m=+376.941783224" Sep 14 01:38:02.584094 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:02.584056 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:38:02.599894 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:02.599868 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:38:03.464279 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:03.464233 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 01:38:57.547951 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.547900 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb"] Sep 14 01:38:57.551293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.551271 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.555155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.554930 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 01:38:57.555155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.554943 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 01:38:57.555155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.555056 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 01:38:57.555475 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.555190 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kpfz9\"" Sep 14 01:38:57.555586 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.555564 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 01:38:57.575157 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.575127 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb"] Sep 14 01:38:57.703452 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.703408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.703452 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.703456 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqwp\" (UniqueName: \"kubernetes.io/projected/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-kube-api-access-4sqwp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.703693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.703547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.703693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.703573 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.703693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.703609 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804055 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.803963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804055 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.804002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804055 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.804027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804055 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.804052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804434 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.804069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqwp\" (UniqueName: \"kubernetes.io/projected/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-kube-api-access-4sqwp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.804568 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.804538 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-manifests\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.806412 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.806392 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-tmp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.806636 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.806614 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-webhook-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.806726 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.806713 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.811478 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.811458 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqwp\" (UniqueName: \"kubernetes.io/projected/7b780b5e-a9f4-4672-8c15-e86b90c37ce6-kube-api-access-4sqwp\") pod \"opendatahub-operator-controller-manager-c5d46b944-7xsrb\" (UID: \"7b780b5e-a9f4-4672-8c15-e86b90c37ce6\") " pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:57.861122 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:57.861094 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:38:58.140748 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:58.140715 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb"] Sep 14 01:38:58.144181 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:38:58.144152 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b780b5e_a9f4_4672_8c15_e86b90c37ce6.slice/crio-66d384736c29cf1fb3e3060069623f531efcc97d1e2544c2850d8ec3dc2dd424 WatchSource:0}: Error finding container 66d384736c29cf1fb3e3060069623f531efcc97d1e2544c2850d8ec3dc2dd424: Status 404 returned error can't find the container with id 66d384736c29cf1fb3e3060069623f531efcc97d1e2544c2850d8ec3dc2dd424 Sep 14 01:38:58.645259 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:38:58.645227 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" event={"ID":"7b780b5e-a9f4-4672-8c15-e86b90c37ce6","Type":"ContainerStarted","Data":"66d384736c29cf1fb3e3060069623f531efcc97d1e2544c2850d8ec3dc2dd424"} Sep 14 01:39:01.661492 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:01.661458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" event={"ID":"7b780b5e-a9f4-4672-8c15-e86b90c37ce6","Type":"ContainerStarted","Data":"3b4694441ab4394d23880dbc2ca8574603c05ce1b86c688aaf1512221ec4ae0d"} Sep 14 01:39:02.667484 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:02.667446 2573 generic.go:356] "Generic (PLEG): container finished" podID="7b780b5e-a9f4-4672-8c15-e86b90c37ce6" containerID="3b4694441ab4394d23880dbc2ca8574603c05ce1b86c688aaf1512221ec4ae0d" exitCode=0 Sep 14 01:39:02.667949 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:02.667537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" event={"ID":"7b780b5e-a9f4-4672-8c15-e86b90c37ce6","Type":"ContainerDied","Data":"3b4694441ab4394d23880dbc2ca8574603c05ce1b86c688aaf1512221ec4ae0d"} Sep 14 01:39:03.673848 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:03.673810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" event={"ID":"7b780b5e-a9f4-4672-8c15-e86b90c37ce6","Type":"ContainerStarted","Data":"e22b6b02e43872dd38a19d5165fb2696cf8ba21eb70e6b4c0dbfffc5536d6eb7"} Sep 14 01:39:03.674276 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:03.673948 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:39:14.680429 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:14.680391 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" Sep 14 01:39:14.702805 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:14.702754 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c5d46b944-7xsrb" podStartSLOduration=13.32801774 podStartE2EDuration="17.70273878s" totalImagesPullingTime="3.320256249s" totalInitContainerRuntime="1.054464791s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:38:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:03.706737041 +0000 UTC m=+490.379853216" watchObservedRunningTime="2026-09-14 01:39:14.70273878 +0000 UTC m=+501.375854954" Sep 14 01:39:36.385209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.385175 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst"] Sep 14 01:39:36.389962 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.389945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.392602 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.392580 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 01:39:36.393662 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.393644 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 01:39:36.393778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.393663 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 01:39:36.393778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.393676 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 01:39:36.393778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.393663 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-68wwc\"" Sep 14 01:39:36.393778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.393644 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 01:39:36.401526 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.401508 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpmx\" (UniqueName: \"kubernetes.io/projected/56725921-913b-4ca7-a9db-5a92fd15a5e1-kube-api-access-zwpmx\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.401603 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.401534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.401603 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.401564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-metrics-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.401603 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.401601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56725921-913b-4ca7-a9db-5a92fd15a5e1-manager-config\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.406889 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.406871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst"] Sep 14 01:39:36.502296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.502245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpmx\" (UniqueName: \"kubernetes.io/projected/56725921-913b-4ca7-a9db-5a92fd15a5e1-kube-api-access-zwpmx\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.502472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.502315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.502472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.502368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-metrics-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.502472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.502396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56725921-913b-4ca7-a9db-5a92fd15a5e1-manager-config\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.503033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.503011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56725921-913b-4ca7-a9db-5a92fd15a5e1-manager-config\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.504920 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.504901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.505020 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.504971 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56725921-913b-4ca7-a9db-5a92fd15a5e1-metrics-cert\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.509725 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.509706 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpmx\" (UniqueName: \"kubernetes.io/projected/56725921-913b-4ca7-a9db-5a92fd15a5e1-kube-api-access-zwpmx\") pod \"lws-controller-manager-5f97d9f44d-46hst\" (UID: \"56725921-913b-4ca7-a9db-5a92fd15a5e1\") " pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.614343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.614309 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz"] Sep 14 01:39:36.617827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.617810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.619995 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.619971 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 14 01:39:36.620110 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.620090 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bj5jq\"" Sep 14 01:39:36.623319 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.623295 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w"] Sep 14 01:39:36.626908 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.626886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.629508 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.629488 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz"] Sep 14 01:39:36.630210 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.630192 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fsn8t\"" Sep 14 01:39:36.630434 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.630308 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 14 01:39:36.639955 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.639895 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w"] Sep 14 01:39:36.698687 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.698659 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:36.703906 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.703885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.703969 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.703929 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcpp\" (UniqueName: \"kubernetes.io/projected/f511c637-3e52-4bf2-9d4b-ebace521fecf-kube-api-access-lhcpp\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.703969 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.703953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58b6s\" (UniqueName: \"kubernetes.io/projected/aafa3390-8519-4cb7-8702-32dfdafbf073-kube-api-access-58b6s\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.704065 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.704049 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.804554 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.804526 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcpp\" (UniqueName: \"kubernetes.io/projected/f511c637-3e52-4bf2-9d4b-ebace521fecf-kube-api-access-lhcpp\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.804730 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.804559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58b6s\" (UniqueName: \"kubernetes.io/projected/aafa3390-8519-4cb7-8702-32dfdafbf073-kube-api-access-58b6s\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.804829 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.804812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.804889 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.804856 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.804961 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:39:36.804943 2573 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 14 01:39:36.805012 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:39:36.804974 2573 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 14 01:39:36.805058 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:39:36.805018 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert podName:aafa3390-8519-4cb7-8702-32dfdafbf073 nodeName:}" failed. No retries permitted until 2026-09-14 01:39:37.304995337 +0000 UTC m=+523.978111523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert") pod "kserve-controller-manager-d6f4b65b8-lzzmz" (UID: "aafa3390-8519-4cb7-8702-32dfdafbf073") : secret "kserve-webhook-server-cert" not found Sep 14 01:39:36.805058 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:39:36.805034 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert podName:f511c637-3e52-4bf2-9d4b-ebace521fecf nodeName:}" failed. No retries permitted until 2026-09-14 01:39:37.305027661 +0000 UTC m=+523.978143815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert") pod "llmisvc-controller-manager-777c6bd7d5-gwc5w" (UID: "f511c637-3e52-4bf2-9d4b-ebace521fecf") : secret "llmisvc-webhook-server-cert" not found Sep 14 01:39:36.816982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.816951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcpp\" (UniqueName: \"kubernetes.io/projected/f511c637-3e52-4bf2-9d4b-ebace521fecf-kube-api-access-lhcpp\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:36.817101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.817018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58b6s\" (UniqueName: \"kubernetes.io/projected/aafa3390-8519-4cb7-8702-32dfdafbf073-kube-api-access-58b6s\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:36.969392 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:36.969369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst"] Sep 14 01:39:36.971209 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:39:36.971180 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56725921_913b_4ca7_a9db_5a92fd15a5e1.slice/crio-974c63a4befd4e94f2655972ff983ad20b65a48f791f19a621c3ed3867988f9b WatchSource:0}: Error finding container 974c63a4befd4e94f2655972ff983ad20b65a48f791f19a621c3ed3867988f9b: Status 404 returned error can't find the container with id 974c63a4befd4e94f2655972ff983ad20b65a48f791f19a621c3ed3867988f9b Sep 14 01:39:37.308716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.308636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:37.308716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.308677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:37.311436 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.311411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f511c637-3e52-4bf2-9d4b-ebace521fecf-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwc5w\" (UID: \"f511c637-3e52-4bf2-9d4b-ebace521fecf\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:37.311557 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.311411 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aafa3390-8519-4cb7-8702-32dfdafbf073-cert\") pod \"kserve-controller-manager-d6f4b65b8-lzzmz\" (UID: \"aafa3390-8519-4cb7-8702-32dfdafbf073\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:37.530689 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.530661 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:37.540216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.540185 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:37.806873 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.806818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" event={"ID":"56725921-913b-4ca7-a9db-5a92fd15a5e1","Type":"ContainerStarted","Data":"974c63a4befd4e94f2655972ff983ad20b65a48f791f19a621c3ed3867988f9b"} Sep 14 01:39:37.843832 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.843804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w"] Sep 14 01:39:37.848802 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:37.848770 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz"] Sep 14 01:39:37.850641 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:39:37.850612 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf511c637_3e52_4bf2_9d4b_ebace521fecf.slice/crio-acf086977c1b1620c5fb652071e0f71295cc6f10afea2bd5a05c3c53eac7298c WatchSource:0}: Error finding container acf086977c1b1620c5fb652071e0f71295cc6f10afea2bd5a05c3c53eac7298c: Status 404 returned error can't find the container with id acf086977c1b1620c5fb652071e0f71295cc6f10afea2bd5a05c3c53eac7298c Sep 14 01:39:37.851331 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:39:37.851310 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafa3390_8519_4cb7_8702_32dfdafbf073.slice/crio-7315579fe5cc524775b05e926dcdd9ec371f4219499c57b92623bb1ff6eebaa3 WatchSource:0}: Error finding container 7315579fe5cc524775b05e926dcdd9ec371f4219499c57b92623bb1ff6eebaa3: Status 404 returned error can't find the container with id 7315579fe5cc524775b05e926dcdd9ec371f4219499c57b92623bb1ff6eebaa3 Sep 14 01:39:38.814763 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:38.814724 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" event={"ID":"f511c637-3e52-4bf2-9d4b-ebace521fecf","Type":"ContainerStarted","Data":"acf086977c1b1620c5fb652071e0f71295cc6f10afea2bd5a05c3c53eac7298c"} Sep 14 01:39:38.816500 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:38.816469 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" event={"ID":"aafa3390-8519-4cb7-8702-32dfdafbf073","Type":"ContainerStarted","Data":"7315579fe5cc524775b05e926dcdd9ec371f4219499c57b92623bb1ff6eebaa3"} Sep 14 01:39:41.834101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.834058 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" event={"ID":"56725921-913b-4ca7-a9db-5a92fd15a5e1","Type":"ContainerStarted","Data":"f5248e859832b33d0589e20189219dc963e6ebecc70c07b36e707ba49d23b290"} Sep 14 01:39:41.834624 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.834172 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:39:41.836201 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.836169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" event={"ID":"aafa3390-8519-4cb7-8702-32dfdafbf073","Type":"ContainerStarted","Data":"9b194e1bb633ae42d89f62e17ecf8a9e01f7bec9e8fbc031cfa1d68f74e28789"} Sep 14 01:39:41.836349 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.836288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:39:41.852316 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.852268 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" podStartSLOduration=1.517263418 podStartE2EDuration="5.852235215s" totalImagesPullingTime="4.334971797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:41.850551207 +0000 UTC m=+528.523667385" watchObservedRunningTime="2026-09-14 01:39:41.852235215 +0000 UTC m=+528.525351390" Sep 14 01:39:41.866498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:41.866454 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" podStartSLOduration=2.411069669 podStartE2EDuration="5.866440964s" totalImagesPullingTime="3.455371295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:41.865217709 +0000 UTC m=+528.538333883" watchObservedRunningTime="2026-09-14 01:39:41.866440964 +0000 UTC m=+528.539557139" Sep 14 01:39:43.845296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:43.845224 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" event={"ID":"f511c637-3e52-4bf2-9d4b-ebace521fecf","Type":"ContainerStarted","Data":"77b6aba14af3412945b7b1a294ec214d0eccdfd17a4d778adf841e6d844c0999"} Sep 14 01:39:43.845654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:43.845306 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:39:43.861775 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:43.861724 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" podStartSLOduration=2.883596855 podStartE2EDuration="7.861705029s" totalImagesPullingTime="4.978108174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:39:43.860206199 +0000 UTC m=+530.533322378" watchObservedRunningTime="2026-09-14 01:39:43.861705029 +0000 UTC m=+530.534821207" Sep 14 01:39:52.842369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:39:52.842293 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f97d9f44d-46hst" Sep 14 01:40:12.845309 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:12.845276 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lzzmz" Sep 14 01:40:14.851470 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:14.851443 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwc5w" Sep 14 01:40:47.547754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.547720 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c"] Sep 14 01:40:47.550971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.550956 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:47.553820 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.553794 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 01:40:47.553820 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.553805 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 14 01:40:47.554030 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.553913 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-24ddg\"" Sep 14 01:40:47.554936 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.554918 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 01:40:47.561237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.561219 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c"] Sep 14 01:40:47.703845 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.703810 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdvdr\" (UniqueName: \"kubernetes.io/projected/3f340ab1-9451-42ec-8bc5-fd9834f68b5d-kube-api-access-vdvdr\") pod \"dns-operator-controller-manager-7b84cd7c9c-pff8c\" (UID: \"3f340ab1-9451-42ec-8bc5-fd9834f68b5d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:47.805290 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.805175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdvdr\" (UniqueName: \"kubernetes.io/projected/3f340ab1-9451-42ec-8bc5-fd9834f68b5d-kube-api-access-vdvdr\") pod \"dns-operator-controller-manager-7b84cd7c9c-pff8c\" (UID: \"3f340ab1-9451-42ec-8bc5-fd9834f68b5d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:47.813918 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.813893 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdvdr\" (UniqueName: \"kubernetes.io/projected/3f340ab1-9451-42ec-8bc5-fd9834f68b5d-kube-api-access-vdvdr\") pod \"dns-operator-controller-manager-7b84cd7c9c-pff8c\" (UID: \"3f340ab1-9451-42ec-8bc5-fd9834f68b5d\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:47.861234 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:47.861201 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:48.132834 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:48.132803 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c"] Sep 14 01:40:48.135664 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:40:48.135638 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f340ab1_9451_42ec_8bc5_fd9834f68b5d.slice/crio-696749a28a24bae9a3cbcea0df7c6569b3a7b11f39bb6f023174143e338d8824 WatchSource:0}: Error finding container 696749a28a24bae9a3cbcea0df7c6569b3a7b11f39bb6f023174143e338d8824: Status 404 returned error can't find the container with id 696749a28a24bae9a3cbcea0df7c6569b3a7b11f39bb6f023174143e338d8824 Sep 14 01:40:49.109414 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:49.109381 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" event={"ID":"3f340ab1-9451-42ec-8bc5-fd9834f68b5d","Type":"ContainerStarted","Data":"696749a28a24bae9a3cbcea0df7c6569b3a7b11f39bb6f023174143e338d8824"} Sep 14 01:40:49.863562 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:49.863522 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx"] Sep 14 01:40:49.868438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:49.868410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:49.872708 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:49.872519 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d7kvr\"" Sep 14 01:40:49.878950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:49.878870 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx"] Sep 14 01:40:50.024379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:50.024325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrh9\" (UniqueName: \"kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9\") pod \"limitador-operator-controller-manager-74495c69f8-ktvjx\" (UID: \"9c112c68-3d06-4356-a373-62e39724de93\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:50.125026 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:50.124945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrh9\" (UniqueName: \"kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9\") pod \"limitador-operator-controller-manager-74495c69f8-ktvjx\" (UID: \"9c112c68-3d06-4356-a373-62e39724de93\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:50.135396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:50.135369 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrh9\" (UniqueName: \"kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9\") pod \"limitador-operator-controller-manager-74495c69f8-ktvjx\" (UID: \"9c112c68-3d06-4356-a373-62e39724de93\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:50.182554 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:50.182515 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:50.817820 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:50.817795 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx"] Sep 14 01:40:50.820435 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:40:50.820410 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c112c68_3d06_4356_a373_62e39724de93.slice/crio-9a34b7e37f95a2f41bd6b8c78b1cea538360cb1654c020fd475bd20b0839bcfc WatchSource:0}: Error finding container 9a34b7e37f95a2f41bd6b8c78b1cea538360cb1654c020fd475bd20b0839bcfc: Status 404 returned error can't find the container with id 9a34b7e37f95a2f41bd6b8c78b1cea538360cb1654c020fd475bd20b0839bcfc Sep 14 01:40:51.121004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:51.120925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" event={"ID":"9c112c68-3d06-4356-a373-62e39724de93","Type":"ContainerStarted","Data":"9a34b7e37f95a2f41bd6b8c78b1cea538360cb1654c020fd475bd20b0839bcfc"} Sep 14 01:40:52.127515 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:52.127476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" event={"ID":"3f340ab1-9451-42ec-8bc5-fd9834f68b5d","Type":"ContainerStarted","Data":"162b95b533db3df7ba9d559219a22c7c8ea271603e7080ef932c48fe7d77313c"} Sep 14 01:40:52.127966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:52.127666 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:40:52.145623 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:52.145574 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" podStartSLOduration=2.040113125 podStartE2EDuration="5.145558038s" totalImagesPullingTime="3.105444913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:40:52.144611572 +0000 UTC m=+598.817727751" watchObservedRunningTime="2026-09-14 01:40:52.145558038 +0000 UTC m=+598.818674212" Sep 14 01:40:55.141511 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:55.141480 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" event={"ID":"9c112c68-3d06-4356-a373-62e39724de93","Type":"ContainerStarted","Data":"32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7"} Sep 14 01:40:55.141915 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:55.141623 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:40:55.156593 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:40:55.156539 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" podStartSLOduration=2.84642582 podStartE2EDuration="6.156519217s" totalImagesPullingTime="3.310093397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:40:55.156018772 +0000 UTC m=+601.829134947" watchObservedRunningTime="2026-09-14 01:40:55.156519217 +0000 UTC m=+601.829635392" Sep 14 01:41:03.135114 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:03.135081 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-pff8c" Sep 14 01:41:04.521637 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.521600 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx"] Sep 14 01:41:04.522093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.521868 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" podUID="9c112c68-3d06-4356-a373-62e39724de93" containerName="manager" containerID="cri-o://32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7" gracePeriod=2 Sep 14 01:41:04.523776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.523754 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:04.529299 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.529273 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx"] Sep 14 01:41:04.532564 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:04.532531 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ktvjx\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="9c112c68-3d06-4356-a373-62e39724de93" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:04.547218 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.547193 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz"] Sep 14 01:41:04.547666 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.547651 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c112c68-3d06-4356-a373-62e39724de93" containerName="manager" Sep 14 01:41:04.547708 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.547670 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c112c68-3d06-4356-a373-62e39724de93" containerName="manager" Sep 14 01:41:04.547761 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.547751 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c112c68-3d06-4356-a373-62e39724de93" containerName="manager" Sep 14 01:41:04.550847 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.550829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:04.552958 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:04.552935 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ktvjx\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="9c112c68-3d06-4356-a373-62e39724de93" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:04.568692 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.568661 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz"] Sep 14 01:41:04.656040 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.656006 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm98h\" (UniqueName: \"kubernetes.io/projected/3e6a1354-3d68-4975-b432-a3e1e84b0330-kube-api-access-mm98h\") pod \"limitador-operator-controller-manager-74495c69f8-4m8cz\" (UID: \"3e6a1354-3d68-4975-b432-a3e1e84b0330\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:04.756596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.756567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm98h\" (UniqueName: \"kubernetes.io/projected/3e6a1354-3d68-4975-b432-a3e1e84b0330-kube-api-access-mm98h\") pod \"limitador-operator-controller-manager-74495c69f8-4m8cz\" (UID: \"3e6a1354-3d68-4975-b432-a3e1e84b0330\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:04.764817 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.764782 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm98h\" (UniqueName: \"kubernetes.io/projected/3e6a1354-3d68-4975-b432-a3e1e84b0330-kube-api-access-mm98h\") pod \"limitador-operator-controller-manager-74495c69f8-4m8cz\" (UID: \"3e6a1354-3d68-4975-b432-a3e1e84b0330\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:04.767142 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.767127 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:04.769435 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:04.769400 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ktvjx\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="9c112c68-3d06-4356-a373-62e39724de93" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:04.857427 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.857340 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwrh9\" (UniqueName: \"kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9\") pod \"9c112c68-3d06-4356-a373-62e39724de93\" (UID: \"9c112c68-3d06-4356-a373-62e39724de93\") " Sep 14 01:41:04.859742 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.859720 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9" (OuterVolumeSpecName: "kube-api-access-kwrh9") pod "9c112c68-3d06-4356-a373-62e39724de93" (UID: "9c112c68-3d06-4356-a373-62e39724de93"). InnerVolumeSpecName "kube-api-access-kwrh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:41:04.896061 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.896035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:04.958816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:04.958787 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrh9\" (UniqueName: \"kubernetes.io/projected/9c112c68-3d06-4356-a373-62e39724de93-kube-api-access-kwrh9\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:41:05.165565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.165542 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz"] Sep 14 01:41:05.167628 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:41:05.167608 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6a1354_3d68_4975_b432_a3e1e84b0330.slice/crio-90efd4a6d228f4713092db3677c44948e37acac83756a07edca06d8824b2b751 WatchSource:0}: Error finding container 90efd4a6d228f4713092db3677c44948e37acac83756a07edca06d8824b2b751: Status 404 returned error can't find the container with id 90efd4a6d228f4713092db3677c44948e37acac83756a07edca06d8824b2b751 Sep 14 01:41:05.183925 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.183902 2573 generic.go:356] "Generic (PLEG): container finished" podID="9c112c68-3d06-4356-a373-62e39724de93" containerID="32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7" exitCode=0 Sep 14 01:41:05.184022 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.183949 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:05.184022 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.183976 2573 scope.go:117] "RemoveContainer" containerID="32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7" Sep 14 01:41:05.185521 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.185495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" event={"ID":"3e6a1354-3d68-4975-b432-a3e1e84b0330","Type":"ContainerStarted","Data":"90efd4a6d228f4713092db3677c44948e37acac83756a07edca06d8824b2b751"} Sep 14 01:41:05.186106 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:05.186078 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ktvjx\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="9c112c68-3d06-4356-a373-62e39724de93" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:05.194350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.194333 2573 scope.go:117] "RemoveContainer" containerID="32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7" Sep 14 01:41:05.194598 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:05.194578 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7\": container with ID starting with 32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7 not found: ID does not exist" containerID="32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7" Sep 14 01:41:05.194645 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.194607 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7"} err="failed to get container status \"32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7\": rpc error: code = NotFound desc = could not find container \"32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7\": container with ID starting with 32226e6e577c50c9902141ebb35e1f9161e0eadce7740dd3376eb0b0850d31b7 not found: ID does not exist" Sep 14 01:41:05.209379 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:41:05.209352 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-ktvjx\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="9c112c68-3d06-4356-a373-62e39724de93" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-ktvjx" Sep 14 01:41:05.861198 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:05.861164 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c112c68-3d06-4356-a373-62e39724de93" path="/var/lib/kubelet/pods/9c112c68-3d06-4356-a373-62e39724de93/volumes" Sep 14 01:41:06.192651 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:06.192620 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" event={"ID":"3e6a1354-3d68-4975-b432-a3e1e84b0330","Type":"ContainerStarted","Data":"a9188fb499117277153142cfe41c59f1e54002c9900fdbf6a255aeef24b69e67"} Sep 14 01:41:06.192803 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:06.192717 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:06.208812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:06.208756 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" podStartSLOduration=2.208740363 podStartE2EDuration="2.208740363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:06.207121746 +0000 UTC m=+612.880237923" watchObservedRunningTime="2026-09-14 01:41:06.208740363 +0000 UTC m=+612.881856538" Sep 14 01:41:17.199907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:17.199834 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4m8cz" Sep 14 01:41:32.846569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.846537 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9"] Sep 14 01:41:32.852044 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.852021 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.854542 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.854516 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 01:41:32.854675 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.854571 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 01:41:32.854675 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.854642 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dtdhr\"" Sep 14 01:41:32.854773 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.854726 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 01:41:32.861417 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.861396 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9"] Sep 14 01:41:32.878008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.877985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878130 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5732174-f596-4ad3-9d33-105200595126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878130 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878044 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878130 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878075 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878355 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rf2t\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-kube-api-access-5rf2t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878390 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5732174-f596-4ad3-9d33-105200595126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.878390 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.878379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.978844 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.978809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.978892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.978951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.978972 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979004 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.978993 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rf2t\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-kube-api-access-5rf2t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979189 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979029 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5732174-f596-4ad3-9d33-105200595126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979189 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979189 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979189 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5732174-f596-4ad3-9d33-105200595126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979231 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979309 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979367 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979549 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979530 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.979663 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.979640 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5732174-f596-4ad3-9d33-105200595126-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.981634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.981612 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5732174-f596-4ad3-9d33-105200595126-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.981738 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.981720 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5732174-f596-4ad3-9d33-105200595126-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.987323 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.987301 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:32.987441 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:32.987420 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rf2t\" (UniqueName: \"kubernetes.io/projected/b5732174-f596-4ad3-9d33-105200595126-kube-api-access-5rf2t\") pod \"maas-default-gateway-openshift-default-78bb78ff78-cqcn9\" (UID: \"b5732174-f596-4ad3-9d33-105200595126\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:33.164093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:33.164061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:33.438500 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:33.438378 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9"] Sep 14 01:41:33.441219 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:41:33.441193 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5732174_f596_4ad3_9d33_105200595126.slice/crio-c83ccaa1b02299a605da9f4247dfd19db53d0d87dc5efd0ab9e70ce7d018eeb9 WatchSource:0}: Error finding container c83ccaa1b02299a605da9f4247dfd19db53d0d87dc5efd0ab9e70ce7d018eeb9: Status 404 returned error can't find the container with id c83ccaa1b02299a605da9f4247dfd19db53d0d87dc5efd0ab9e70ce7d018eeb9 Sep 14 01:41:33.443266 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:33.443237 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:41:34.317865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:34.317821 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" event={"ID":"b5732174-f596-4ad3-9d33-105200595126","Type":"ContainerStarted","Data":"c83ccaa1b02299a605da9f4247dfd19db53d0d87dc5efd0ab9e70ce7d018eeb9"} Sep 14 01:41:36.258114 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:36.258070 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:41:36.258558 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:36.258149 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:41:36.258558 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:36.258184 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:41:36.330212 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:36.330185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" event={"ID":"b5732174-f596-4ad3-9d33-105200595126","Type":"ContainerStarted","Data":"91d8168111fa58e134f326a0562f5415899c0576b209f638c2e5aa0d0e2182ba"} Sep 14 01:41:36.347724 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:36.347668 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podStartSLOduration=1.5332420340000001 podStartE2EDuration="4.347646592s" totalImagesPullingTime="2.814404558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:36.346765064 +0000 UTC m=+643.019881243" watchObservedRunningTime="2026-09-14 01:41:36.347646592 +0000 UTC m=+643.020762769" Sep 14 01:41:37.165038 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:37.164994 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:37.169902 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:37.169879 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:37.334389 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:37.334356 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:37.335231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:37.335210 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:41:46.245965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.245929 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:41:46.249363 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.249341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.252515 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.252499 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2nlnr\"" Sep 14 01:41:46.252580 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.252499 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 14 01:41:46.258285 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.258242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:41:46.346501 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.346468 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:41:46.392645 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.392618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.392821 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.392663 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmghn\" (UniqueName: \"kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.494165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.494132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.494353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.494178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmghn\" (UniqueName: \"kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.494772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.494753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.502619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.502561 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmghn\" (UniqueName: \"kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn\") pod \"limitador-limitador-8588749977-dvv9d\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.560571 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.560525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:46.831759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:46.831733 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:41:46.834483 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:41:46.834457 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7f5738_f753_4c51_b6df_de0345dec5e9.slice/crio-f70692a763311aec9e86b8544b283001fd8d0a5080bcd847e9a0e5abddb7fca7 WatchSource:0}: Error finding container f70692a763311aec9e86b8544b283001fd8d0a5080bcd847e9a0e5abddb7fca7: Status 404 returned error can't find the container with id f70692a763311aec9e86b8544b283001fd8d0a5080bcd847e9a0e5abddb7fca7 Sep 14 01:41:47.376549 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:47.376516 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" event={"ID":"4e7f5738-f753-4c51-b6df-de0345dec5e9","Type":"ContainerStarted","Data":"f70692a763311aec9e86b8544b283001fd8d0a5080bcd847e9a0e5abddb7fca7"} Sep 14 01:41:48.382579 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:48.382552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" event={"ID":"4e7f5738-f753-4c51-b6df-de0345dec5e9","Type":"ContainerStarted","Data":"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c"} Sep 14 01:41:48.382938 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:48.382746 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:41:48.398807 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:48.398762 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" podStartSLOduration=1.129824904 podStartE2EDuration="2.398746428s" totalImagesPullingTime="1.268921524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:41:48.3970362 +0000 UTC m=+655.070152376" watchObservedRunningTime="2026-09-14 01:41:48.398746428 +0000 UTC m=+655.071862602" Sep 14 01:41:59.388781 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:41:59.388753 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:42:01.887388 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:01.887355 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:42:01.887800 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:01.887525 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" podUID="4e7f5738-f753-4c51-b6df-de0345dec5e9" containerName="limitador" containerID="cri-o://a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c" gracePeriod=30 Sep 14 01:42:02.435121 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.435097 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:42:02.443299 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.443247 2573 generic.go:356] "Generic (PLEG): container finished" podID="4e7f5738-f753-4c51-b6df-de0345dec5e9" containerID="a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c" exitCode=0 Sep 14 01:42:02.443410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.443305 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" Sep 14 01:42:02.443410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.443351 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" event={"ID":"4e7f5738-f753-4c51-b6df-de0345dec5e9","Type":"ContainerDied","Data":"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c"} Sep 14 01:42:02.443410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.443396 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-dvv9d" event={"ID":"4e7f5738-f753-4c51-b6df-de0345dec5e9","Type":"ContainerDied","Data":"f70692a763311aec9e86b8544b283001fd8d0a5080bcd847e9a0e5abddb7fca7"} Sep 14 01:42:02.443515 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.443418 2573 scope.go:117] "RemoveContainer" containerID="a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c" Sep 14 01:42:02.452571 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.452546 2573 scope.go:117] "RemoveContainer" containerID="a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c" Sep 14 01:42:02.452919 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:02.452899 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c\": container with ID starting with a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c not found: ID does not exist" containerID="a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c" Sep 14 01:42:02.453016 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.452931 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c"} err="failed to get container status \"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c\": rpc error: code = NotFound desc = could not find container \"a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c\": container with ID starting with a397f8045456303a591817358cf6eec0ee51732e1b751ca47bb27ce34370f41c not found: ID does not exist" Sep 14 01:42:02.534973 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.534939 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file\") pod \"4e7f5738-f753-4c51-b6df-de0345dec5e9\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " Sep 14 01:42:02.535160 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.535036 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmghn\" (UniqueName: \"kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn\") pod \"4e7f5738-f753-4c51-b6df-de0345dec5e9\" (UID: \"4e7f5738-f753-4c51-b6df-de0345dec5e9\") " Sep 14 01:42:02.535376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.535353 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file" (OuterVolumeSpecName: "config-file") pod "4e7f5738-f753-4c51-b6df-de0345dec5e9" (UID: "4e7f5738-f753-4c51-b6df-de0345dec5e9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:42:02.537244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.537213 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn" (OuterVolumeSpecName: "kube-api-access-rmghn") pod "4e7f5738-f753-4c51-b6df-de0345dec5e9" (UID: "4e7f5738-f753-4c51-b6df-de0345dec5e9"). InnerVolumeSpecName "kube-api-access-rmghn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:42:02.636614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.636583 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e7f5738-f753-4c51-b6df-de0345dec5e9-config-file\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:42:02.636614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.636610 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmghn\" (UniqueName: \"kubernetes.io/projected/4e7f5738-f753-4c51-b6df-de0345dec5e9-kube-api-access-rmghn\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:42:02.764526 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.764491 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:42:02.773224 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:02.773194 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-dvv9d"] Sep 14 01:42:03.860972 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:03.860932 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7f5738-f753-4c51-b6df-de0345dec5e9" path="/var/lib/kubelet/pods/4e7f5738-f753-4c51-b6df-de0345dec5e9/volumes" Sep 14 01:42:04.047440 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.047410 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc"] Sep 14 01:42:04.047729 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.047716 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7f5738-f753-4c51-b6df-de0345dec5e9" containerName="limitador" Sep 14 01:42:04.047729 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.047730 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7f5738-f753-4c51-b6df-de0345dec5e9" containerName="limitador" Sep 14 01:42:04.047850 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.047794 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7f5738-f753-4c51-b6df-de0345dec5e9" containerName="limitador" Sep 14 01:42:04.052238 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.052211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.054667 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.054643 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 01:42:04.054783 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.054648 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 14 01:42:04.055110 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.055092 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 01:42:04.055187 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.055151 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-j5mpq\"" Sep 14 01:42:04.058391 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.058373 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc"] Sep 14 01:42:04.151726 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.151693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e054f313-1f8d-44b8-a5b9-0d592a882809-data\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.151901 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.151814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzb7\" (UniqueName: \"kubernetes.io/projected/e054f313-1f8d-44b8-a5b9-0d592a882809-kube-api-access-2zzb7\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.252695 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.252644 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e054f313-1f8d-44b8-a5b9-0d592a882809-data\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.252879 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.252748 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzb7\" (UniqueName: \"kubernetes.io/projected/e054f313-1f8d-44b8-a5b9-0d592a882809-kube-api-access-2zzb7\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.253040 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.253021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e054f313-1f8d-44b8-a5b9-0d592a882809-data\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.260874 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.260844 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzb7\" (UniqueName: \"kubernetes.io/projected/e054f313-1f8d-44b8-a5b9-0d592a882809-kube-api-access-2zzb7\") pod \"postgres-69fb4594cf-zs8vc\" (UID: \"e054f313-1f8d-44b8-a5b9-0d592a882809\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.363997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.363959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:04.659645 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:04.659611 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc"] Sep 14 01:42:04.663383 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:42:04.663357 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode054f313_1f8d_44b8_a5b9_0d592a882809.slice/crio-64c46e026d7102168ab67e87930eb7b11cb9aba858bb081e0de3fa2e391818d1 WatchSource:0}: Error finding container 64c46e026d7102168ab67e87930eb7b11cb9aba858bb081e0de3fa2e391818d1: Status 404 returned error can't find the container with id 64c46e026d7102168ab67e87930eb7b11cb9aba858bb081e0de3fa2e391818d1 Sep 14 01:42:05.463207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:05.463121 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" event={"ID":"e054f313-1f8d-44b8-a5b9-0d592a882809","Type":"ContainerStarted","Data":"64c46e026d7102168ab67e87930eb7b11cb9aba858bb081e0de3fa2e391818d1"} Sep 14 01:42:09.489486 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:09.489454 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" event={"ID":"e054f313-1f8d-44b8-a5b9-0d592a882809","Type":"ContainerStarted","Data":"e6ddc0f8d714e92baf473728d102d5b68a2902bfa38e0dfd61ed08467f01a80d"} Sep 14 01:42:09.489882 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:09.489546 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:09.504395 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:09.504332 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" podStartSLOduration=0.78354746 podStartE2EDuration="5.504318184s" totalImagesPullingTime="4.720770724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:42:09.502690279 +0000 UTC m=+676.175806456" watchObservedRunningTime="2026-09-14 01:42:09.504318184 +0000 UTC m=+676.177434359" Sep 14 01:42:15.523537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:15.523500 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zs8vc" Sep 14 01:42:22.545414 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.545380 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:22.550179 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.550157 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.552688 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.552669 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 01:42:22.552805 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.552732 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 01:42:22.552805 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.552669 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 01:42:22.552805 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.552768 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hchhd\"" Sep 14 01:42:22.561137 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.561115 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:22.616835 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.616806 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.616976 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.616857 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.616976 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.616917 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4xf\" (UniqueName: \"kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.716067 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.716036 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 01:42:22.717439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.717409 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.717556 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.717454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.717556 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.717482 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4xf\" (UniqueName: \"kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.717634 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:22.717563 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 14 01:42:22.717634 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:22.717624 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs podName:0e698576-65c8-4ca6-8800-73076a0d0f2a nodeName:}" failed. No retries permitted until 2026-09-14 01:42:23.217607755 +0000 UTC m=+689.890723914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs") pod "maas-controller-b5d9d84b8-plzw5" (UID: "0e698576-65c8-4ca6-8800-73076a0d0f2a") : secret "maas-controller-metrics-tls" not found Sep 14 01:42:22.719908 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.719884 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.720442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.720420 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.728524 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.728496 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4xf\" (UniqueName: \"kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:22.730462 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.730437 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 01:42:22.818057 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.817980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4bq\" (UniqueName: \"kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.818057 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.818020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.818239 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.818079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.919204 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.919168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.919367 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.919234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4bq\" (UniqueName: \"kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.919367 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.919295 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.919367 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:22.919345 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 14 01:42:22.919481 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:22.919405 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs podName:f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e nodeName:}" failed. No retries permitted until 2026-09-14 01:42:23.419389619 +0000 UTC m=+690.092505772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs") pod "maas-controller-59dfd7d9fd-8sfc2" (UID: "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e") : secret "maas-controller-metrics-tls" not found Sep 14 01:42:22.921823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.921805 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:22.928370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:22.928346 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4bq\" (UniqueName: \"kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:23.222349 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:23.222316 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:23.222524 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:23.222436 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 14 01:42:23.222524 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:23.222486 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs podName:0e698576-65c8-4ca6-8800-73076a0d0f2a nodeName:}" failed. No retries permitted until 2026-09-14 01:42:24.222474392 +0000 UTC m=+690.895590545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs") pod "maas-controller-b5d9d84b8-plzw5" (UID: "0e698576-65c8-4ca6-8800-73076a0d0f2a") : secret "maas-controller-metrics-tls" not found Sep 14 01:42:23.424385 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:23.424349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:23.426863 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:23.426842 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") pod \"maas-controller-59dfd7d9fd-8sfc2\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:23.643993 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:23.643965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:23.914120 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:23.914030 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 01:42:23.916704 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:42:23.916675 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fd70fd_0a7e_44ae_80f1_68c5dd1f1e3e.slice/crio-6acd40d3c0cc5d8caaeb9c7e715d941e1456f40413fc05c74275c7c63c7092ff WatchSource:0}: Error finding container 6acd40d3c0cc5d8caaeb9c7e715d941e1456f40413fc05c74275c7c63c7092ff: Status 404 returned error can't find the container with id 6acd40d3c0cc5d8caaeb9c7e715d941e1456f40413fc05c74275c7c63c7092ff Sep 14 01:42:24.231347 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:24.231245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:24.233753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:24.233723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"maas-controller-b5d9d84b8-plzw5\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:24.361377 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:24.361343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:24.559121 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:24.559036 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" event={"ID":"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e","Type":"ContainerStarted","Data":"6acd40d3c0cc5d8caaeb9c7e715d941e1456f40413fc05c74275c7c63c7092ff"} Sep 14 01:42:24.636126 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:24.636092 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:24.639599 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:42:24.639562 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e698576_65c8_4ca6_8800_73076a0d0f2a.slice/crio-de7a44c2763cbce4dd407d6fd6ed8c956eb85274f0377d09ef2e762b55eea43e WatchSource:0}: Error finding container de7a44c2763cbce4dd407d6fd6ed8c956eb85274f0377d09ef2e762b55eea43e: Status 404 returned error can't find the container with id de7a44c2763cbce4dd407d6fd6ed8c956eb85274f0377d09ef2e762b55eea43e Sep 14 01:42:25.566679 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:25.566645 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" event={"ID":"0e698576-65c8-4ca6-8800-73076a0d0f2a","Type":"ContainerStarted","Data":"de7a44c2763cbce4dd407d6fd6ed8c956eb85274f0377d09ef2e762b55eea43e"} Sep 14 01:42:26.573778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.573740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" event={"ID":"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e","Type":"ContainerStarted","Data":"aec44580aced7dbaedbdea2d5b5016918f2c695abc7ac7ff0e1aee08e3d9a8f6"} Sep 14 01:42:26.574341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.573858 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:26.575539 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.575514 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" event={"ID":"0e698576-65c8-4ca6-8800-73076a0d0f2a","Type":"ContainerStarted","Data":"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e"} Sep 14 01:42:26.575681 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.575562 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:26.589674 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.589617 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" podStartSLOduration=2.020072512 podStartE2EDuration="4.589599946s" totalImagesPullingTime="2.569527434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:42:26.588144829 +0000 UTC m=+693.261261017" watchObservedRunningTime="2026-09-14 01:42:26.589599946 +0000 UTC m=+693.262716121" Sep 14 01:42:26.603311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:26.603272 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" podStartSLOduration=2.750526355 podStartE2EDuration="4.603231385s" totalImagesPullingTime="1.85270503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:42:26.602693573 +0000 UTC m=+693.275809753" watchObservedRunningTime="2026-09-14 01:42:26.603231385 +0000 UTC m=+693.276347560" Sep 14 01:42:37.584155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.584126 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:37.584636 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.584197 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 01:42:37.642797 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.642749 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:37.643064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.642982 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" podUID="0e698576-65c8-4ca6-8800-73076a0d0f2a" containerName="manager" containerID="cri-o://574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e" gracePeriod=30 Sep 14 01:42:37.887174 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.887144 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:37.957108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.957077 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert\") pod \"0e698576-65c8-4ca6-8800-73076a0d0f2a\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " Sep 14 01:42:37.957306 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.957124 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") pod \"0e698576-65c8-4ca6-8800-73076a0d0f2a\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " Sep 14 01:42:37.957306 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.957164 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4xf\" (UniqueName: \"kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf\") pod \"0e698576-65c8-4ca6-8800-73076a0d0f2a\" (UID: \"0e698576-65c8-4ca6-8800-73076a0d0f2a\") " Sep 14 01:42:37.959469 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.959437 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0e698576-65c8-4ca6-8800-73076a0d0f2a" (UID: "0e698576-65c8-4ca6-8800-73076a0d0f2a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:42:37.959565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.959471 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e698576-65c8-4ca6-8800-73076a0d0f2a" (UID: "0e698576-65c8-4ca6-8800-73076a0d0f2a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:42:37.959565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:37.959512 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf" (OuterVolumeSpecName: "kube-api-access-kt4xf") pod "0e698576-65c8-4ca6-8800-73076a0d0f2a" (UID: "0e698576-65c8-4ca6-8800-73076a0d0f2a"). InnerVolumeSpecName "kube-api-access-kt4xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:42:38.058459 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.058426 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:42:38.058459 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.058453 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4xf\" (UniqueName: \"kubernetes.io/projected/0e698576-65c8-4ca6-8800-73076a0d0f2a-kube-api-access-kt4xf\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:42:38.058459 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.058464 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e698576-65c8-4ca6-8800-73076a0d0f2a-webhook-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:42:38.633842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.633808 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e698576-65c8-4ca6-8800-73076a0d0f2a" containerID="574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e" exitCode=0 Sep 14 01:42:38.633842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.633847 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" event={"ID":"0e698576-65c8-4ca6-8800-73076a0d0f2a","Type":"ContainerDied","Data":"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e"} Sep 14 01:42:38.634317 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.633864 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" Sep 14 01:42:38.634317 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.633916 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b5d9d84b8-plzw5" event={"ID":"0e698576-65c8-4ca6-8800-73076a0d0f2a","Type":"ContainerDied","Data":"de7a44c2763cbce4dd407d6fd6ed8c956eb85274f0377d09ef2e762b55eea43e"} Sep 14 01:42:38.634317 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.633939 2573 scope.go:117] "RemoveContainer" containerID="574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e" Sep 14 01:42:38.642477 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.642460 2573 scope.go:117] "RemoveContainer" containerID="574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e" Sep 14 01:42:38.642740 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:38.642723 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e\": container with ID starting with 574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e not found: ID does not exist" containerID="574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e" Sep 14 01:42:38.642811 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.642746 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e"} err="failed to get container status \"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e\": rpc error: code = NotFound desc = could not find container \"574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e\": container with ID starting with 574b54c92e57ef41cf652727d0db3f82dd18a82538f4e51f6cd8e0e11c67fa8e not found: ID does not exist" Sep 14 01:42:38.653811 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.653781 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:38.655301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:38.655283 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b5d9d84b8-plzw5"] Sep 14 01:42:39.860891 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:39.860859 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e698576-65c8-4ca6-8800-73076a0d0f2a" path="/var/lib/kubelet/pods/0e698576-65c8-4ca6-8800-73076a0d0f2a/volumes" Sep 14 01:42:59.567263 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.567214 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 01:42:59.567709 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.567578 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e698576-65c8-4ca6-8800-73076a0d0f2a" containerName="manager" Sep 14 01:42:59.567709 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.567590 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e698576-65c8-4ca6-8800-73076a0d0f2a" containerName="manager" Sep 14 01:42:59.567709 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.567649 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e698576-65c8-4ca6-8800-73076a0d0f2a" containerName="manager" Sep 14 01:42:59.573663 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.573644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.576169 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.576140 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 01:42:59.576293 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.576239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 01:42:59.576346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.576295 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hsv9\"" Sep 14 01:42:59.581090 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.581069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 01:42:59.627141 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.627113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plc8b\" (UniqueName: \"kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.627297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.627168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.627297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.627211 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.627297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.627232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.727867 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.727834 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.727867 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.727871 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.728062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.727951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plc8b\" (UniqueName: \"kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.728062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.728010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.728062 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:59.728046 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 14 01:42:59.728173 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:42:59.728097 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs podName:0342fde0-a4ac-4375-aca8-e3122d747a30 nodeName:}" failed. No retries permitted until 2026-09-14 01:43:00.228081828 +0000 UTC m=+726.901197980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs") pod "maas-api-7c69887854-f5knc" (UID: "0342fde0-a4ac-4375-aca8-e3122d747a30") : secret "maas-api-metrics-tls" not found Sep 14 01:42:59.728640 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.728620 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.730503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.730480 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:42:59.741182 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:42:59.741163 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plc8b\" (UniqueName: \"kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:00.066161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.066125 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-c7wsb"] Sep 14 01:43:00.069606 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.069589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.072702 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.072679 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dwfhr\"" Sep 14 01:43:00.072821 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.072679 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 01:43:00.084687 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.084666 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-c7wsb"] Sep 14 01:43:00.130901 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.130870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e827c8-d189-4b67-ae62-6a60e9ce057a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.131038 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.130934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqf8\" (UniqueName: \"kubernetes.io/projected/b3e827c8-d189-4b67-ae62-6a60e9ce057a-kube-api-access-shqf8\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.231831 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.231788 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e827c8-d189-4b67-ae62-6a60e9ce057a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.232015 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.231849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqf8\" (UniqueName: \"kubernetes.io/projected/b3e827c8-d189-4b67-ae62-6a60e9ce057a-kube-api-access-shqf8\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.232015 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.231885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:00.232509 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.232486 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3e827c8-d189-4b67-ae62-6a60e9ce057a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.234396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.234375 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") pod \"maas-api-7c69887854-f5knc\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:00.239522 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.239499 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqf8\" (UniqueName: \"kubernetes.io/projected/b3e827c8-d189-4b67-ae62-6a60e9ce057a-kube-api-access-shqf8\") pod \"payload-processing-5b54b7fbb7-c7wsb\" (UID: \"b3e827c8-d189-4b67-ae62-6a60e9ce057a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.379616 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.379532 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:00.484948 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.484914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:00.650747 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.650722 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-c7wsb"] Sep 14 01:43:00.652481 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:43:00.652455 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e827c8_d189_4b67_ae62_6a60e9ce057a.slice/crio-2b371c5e5acbab39b99768e45a4da8983725e6d5d4b1271bed7ac25f34fb3d2f WatchSource:0}: Error finding container 2b371c5e5acbab39b99768e45a4da8983725e6d5d4b1271bed7ac25f34fb3d2f: Status 404 returned error can't find the container with id 2b371c5e5acbab39b99768e45a4da8983725e6d5d4b1271bed7ac25f34fb3d2f Sep 14 01:43:00.732134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.732101 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" event={"ID":"b3e827c8-d189-4b67-ae62-6a60e9ce057a","Type":"ContainerStarted","Data":"2b371c5e5acbab39b99768e45a4da8983725e6d5d4b1271bed7ac25f34fb3d2f"} Sep 14 01:43:00.761141 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:00.761048 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 01:43:01.741070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:01.741037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" event={"ID":"0342fde0-a4ac-4375-aca8-e3122d747a30","Type":"ContainerStarted","Data":"f41aa3646b065916f8888bd5324b9fd985e4b860c02c3200b87f7d47fcbefc46"} Sep 14 01:43:04.756831 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.756787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" event={"ID":"0342fde0-a4ac-4375-aca8-e3122d747a30","Type":"ContainerStarted","Data":"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a"} Sep 14 01:43:04.757335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.756868 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:04.759027 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.758995 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" event={"ID":"b3e827c8-d189-4b67-ae62-6a60e9ce057a","Type":"ContainerStarted","Data":"aa04c9fe9d8d5f283046d6ee04bfab52df52e0bb56aa81ca21ca22165f9e6174"} Sep 14 01:43:04.759550 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.759524 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:04.773495 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.773446 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" podStartSLOduration=2.410069656 podStartE2EDuration="5.773427511s" totalImagesPullingTime="3.363357855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:04.771948304 +0000 UTC m=+731.445064480" watchObservedRunningTime="2026-09-14 01:43:04.773427511 +0000 UTC m=+731.446543687" Sep 14 01:43:04.787424 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:04.787381 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" podStartSLOduration=1.231184068 podStartE2EDuration="4.787363113s" totalImagesPullingTime="3.556179045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:04.78579752 +0000 UTC m=+731.458913696" watchObservedRunningTime="2026-09-14 01:43:04.787363113 +0000 UTC m=+731.460479294" Sep 14 01:43:08.376414 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:08.376376 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 01:43:08.376810 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:08.376442 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 01:43:09.786354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.786322 2573 generic.go:356] "Generic (PLEG): container finished" podID="b5732174-f596-4ad3-9d33-105200595126" containerID="91d8168111fa58e134f326a0562f5415899c0576b209f638c2e5aa0d0e2182ba" exitCode=0 Sep 14 01:43:09.786747 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.786400 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" event={"ID":"b5732174-f596-4ad3-9d33-105200595126","Type":"ContainerDied","Data":"91d8168111fa58e134f326a0562f5415899c0576b209f638c2e5aa0d0e2182ba"} Sep 14 01:43:09.786799 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.786787 2573 scope.go:117] "RemoveContainer" containerID="91d8168111fa58e134f326a0562f5415899c0576b209f638c2e5aa0d0e2182ba" Sep 14 01:43:09.787785 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.787760 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:43:09.787833 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.787825 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:43:09.787865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:09.787856 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:43:10.775520 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:10.775486 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 01:43:10.793530 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:10.793491 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" event={"ID":"b5732174-f596-4ad3-9d33-105200595126","Type":"ContainerStarted","Data":"c01bfbbffc561d6b32ef8f7d473fc0daa332ecbdf8c6fabd3934ac9dcd0c6f7d"} Sep 14 01:43:11.164812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:11.164753 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:43:11.165165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:11.164869 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 14 01:43:11.165408 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:11.165364 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 14 01:43:12.164820 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:12.164787 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 14 01:43:12.165413 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:12.165382 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 14 01:43:13.164638 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:13.164601 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 14 01:43:13.164906 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:13.164876 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 14 01:43:14.164978 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:14.164930 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 14 01:43:14.165494 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:14.165002 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 14 01:43:15.165314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:15.164814 2573 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-cqcn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Sep 14 01:43:15.165314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:15.164897 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" podUID="b5732174-f596-4ad3-9d33-105200595126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Sep 14 01:43:16.167898 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:16.167864 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:43:16.168343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:16.168269 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:43:16.168892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:16.168873 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-cqcn9" Sep 14 01:43:16.772054 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:16.772021 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-c7wsb" Sep 14 01:43:45.012900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.012870 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf"] Sep 14 01:43:45.020151 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.020129 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.023946 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.023920 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 01:43:45.024519 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.024499 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tpvrt\"" Sep 14 01:43:45.024649 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.024499 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 01:43:45.024716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.024498 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 14 01:43:45.026534 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.026509 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf"] Sep 14 01:43:45.061208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061182 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.061381 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhqs\" (UniqueName: \"kubernetes.io/projected/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-kube-api-access-cmhqs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.061381 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.061381 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.061381 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061365 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.061547 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.061400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162229 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162198 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhqs\" (UniqueName: \"kubernetes.io/projected/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-kube-api-access-cmhqs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162229 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162692 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162669 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162755 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162710 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.162807 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.162759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.164772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.164748 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.164972 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.164951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.170160 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.170140 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhqs\" (UniqueName: \"kubernetes.io/projected/e8088731-9d9e-4f70-9a4e-8f2bf63dfa89-kube-api-access-cmhqs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf\" (UID: \"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.332992 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.332908 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:45.602498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.602443 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf"] Sep 14 01:43:45.604866 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:43:45.604836 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8088731_9d9e_4f70_9a4e_8f2bf63dfa89.slice/crio-a58edbbac88ced89e17f199043220255e928cf383695520d4b28285734cd6115 WatchSource:0}: Error finding container a58edbbac88ced89e17f199043220255e928cf383695520d4b28285734cd6115: Status 404 returned error can't find the container with id a58edbbac88ced89e17f199043220255e928cf383695520d4b28285734cd6115 Sep 14 01:43:45.963120 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:45.963088 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" event={"ID":"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89","Type":"ContainerStarted","Data":"a58edbbac88ced89e17f199043220255e928cf383695520d4b28285734cd6115"} Sep 14 01:43:46.969148 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:46.969066 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" event={"ID":"e8088731-9d9e-4f70-9a4e-8f2bf63dfa89","Type":"ContainerStarted","Data":"89a19d20771bf41f3aa40b4347eb615ec3b50984ecb95c29d116d1f917db97d5"} Sep 14 01:43:46.986611 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:46.986553 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" podStartSLOduration=1.878437158 podStartE2EDuration="2.98653644s" totalImagesPullingTime="1.108099282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:43:46.984429374 +0000 UTC m=+773.657545549" watchObservedRunningTime="2026-09-14 01:43:46.98653644 +0000 UTC m=+773.659652615" Sep 14 01:43:55.333418 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:55.333367 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:55.333841 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:55.333446 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:55.346440 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:55.346413 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:43:56.021190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:43:56.021162 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf" Sep 14 01:44:13.313045 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.313008 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg"] Sep 14 01:44:13.317705 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.317682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.320137 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.320117 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 14 01:44:13.325828 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.325802 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg"] Sep 14 01:44:13.402634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0646d9-48ef-46ac-afec-86329c61c5a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.402634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.402864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.402864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402703 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.402864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.402864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.402768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx9xm\" (UniqueName: \"kubernetes.io/projected/0b0646d9-48ef-46ac-afec-86329c61c5a9-kube-api-access-bx9xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504173 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504173 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx9xm\" (UniqueName: \"kubernetes.io/projected/0b0646d9-48ef-46ac-afec-86329c61c5a9-kube-api-access-bx9xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0646d9-48ef-46ac-afec-86329c61c5a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504647 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504623 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504753 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.504835 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.504784 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.506488 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.506472 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0646d9-48ef-46ac-afec-86329c61c5a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.506809 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.506793 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0646d9-48ef-46ac-afec-86329c61c5a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.512301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.512274 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx9xm\" (UniqueName: \"kubernetes.io/projected/0b0646d9-48ef-46ac-afec-86329c61c5a9-kube-api-access-bx9xm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg\" (UID: \"0b0646d9-48ef-46ac-afec-86329c61c5a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.629438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.629363 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:13.901564 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:13.901539 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg"] Sep 14 01:44:13.904528 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:44:13.904498 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0646d9_48ef_46ac_afec_86329c61c5a9.slice/crio-841037b6945b5bb6c7bbe6c968f200a19ededde90c03ec481715c2d8f563c664 WatchSource:0}: Error finding container 841037b6945b5bb6c7bbe6c968f200a19ededde90c03ec481715c2d8f563c664: Status 404 returned error can't find the container with id 841037b6945b5bb6c7bbe6c968f200a19ededde90c03ec481715c2d8f563c664 Sep 14 01:44:14.092565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:14.092527 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" event={"ID":"0b0646d9-48ef-46ac-afec-86329c61c5a9","Type":"ContainerStarted","Data":"841037b6945b5bb6c7bbe6c968f200a19ededde90c03ec481715c2d8f563c664"} Sep 14 01:44:15.098961 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:15.098878 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" event={"ID":"0b0646d9-48ef-46ac-afec-86329c61c5a9","Type":"ContainerStarted","Data":"d77e7141b98d5ccd4ad6dde564cf8d58a2a01b0d9604eb0d3ed4fb499d1c1862"} Sep 14 01:44:15.115441 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:15.115396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" podStartSLOduration=1.9443215280000001 podStartE2EDuration="2.115379356s" totalImagesPullingTime="171.057828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:44:15.113641175 +0000 UTC m=+801.786757349" watchObservedRunningTime="2026-09-14 01:44:15.115379356 +0000 UTC m=+801.788495531" Sep 14 01:44:23.629957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:23.629912 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:23.630491 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:23.629986 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:23.642491 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:23.642467 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:44:24.151366 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:44:24.151336 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg" Sep 14 01:46:05.231108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.231072 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:46:05.235078 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.235059 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.237438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.237413 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a36bf3-openshift-default-dockercfg-c656k\"" Sep 14 01:46:05.246780 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.246753 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:46:05.295935 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.295901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.295951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.295989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296109 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296127 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pz6w\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.296420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.296220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.397620 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.397814 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.397814 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pz6w\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.397814 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.397814 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.398008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397857 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.398008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397892 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.398008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.398008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.397984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.398898 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.398858 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.399140 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.399121 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.403750 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.403645 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.406490 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.406461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.406596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.406567 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.407003 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.406972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.407092 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.407024 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.408235 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.408215 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pz6w\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.408801 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.408783 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token\") pod \"e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.548564 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.548486 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:05.822346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.822277 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:46:05.824708 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:05.824679 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d120a9_ebf2_459b_a251_5341f9ed383b.slice/crio-06aed4017286e79a1fdb1911d60a3c2966860084797d4828d31bcdd38982e3c8 WatchSource:0}: Error finding container 06aed4017286e79a1fdb1911d60a3c2966860084797d4828d31bcdd38982e3c8: Status 404 returned error can't find the container with id 06aed4017286e79a1fdb1911d60a3c2966860084797d4828d31bcdd38982e3c8 Sep 14 01:46:05.827064 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.827020 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:05.827153 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.827107 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:05.827193 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:05.827148 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:06.625157 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:06.625119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerStarted","Data":"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40"} Sep 14 01:46:06.625529 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:06.625185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerStarted","Data":"06aed4017286e79a1fdb1911d60a3c2966860084797d4828d31bcdd38982e3c8"} Sep 14 01:46:06.647334 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:06.647280 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podStartSLOduration=1.6472460309999999 podStartE2EDuration="1.647246031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:06.644897753 +0000 UTC m=+913.318013929" watchObservedRunningTime="2026-09-14 01:46:06.647246031 +0000 UTC m=+913.320362206" Sep 14 01:46:07.549502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:07.549469 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:07.554641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:07.554617 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:07.629504 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:07.629478 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:07.630461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:07.630440 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:09.623481 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.623447 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:46:09.626951 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.626930 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.629512 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.629490 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-99c616-serving-cert\"" Sep 14 01:46:09.636426 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.636400 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:46:09.638354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.638320 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpc8v\" (UniqueName: \"kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.638460 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.638405 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.638561 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.638540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.638677 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.638582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.739247 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.739207 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.739524 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.739500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.739640 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.739551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpc8v\" (UniqueName: \"kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.739640 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.739612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.739748 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:46:09.739723 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-serving-cert: secret "maas-api-e2e-worker-w0-99c616-serving-cert" not found Sep 14 01:46:09.739808 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:46:09.739784 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls podName:7c44ce5d-4f0a-40d1-8f98-dadee2624fbf nodeName:}" failed. No retries permitted until 2026-09-14 01:46:10.239763452 +0000 UTC m=+916.912879628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls") pod "maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" (UID: "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf") : secret "maas-api-e2e-worker-w0-99c616-serving-cert" not found Sep 14 01:46:09.740353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.740324 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.742302 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.742282 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:09.747836 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:09.747810 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpc8v\" (UniqueName: \"kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:10.244018 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.243974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:10.246631 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.246598 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") pod \"maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:10.540796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.540706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:10.823744 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.823694 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:46:10.826404 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.826368 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 01:46:10.827044 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:10.826700 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c44ce5d_4f0a_40d1_8f98_dadee2624fbf.slice/crio-a83257c0942addf97feed1f193bf391fbf35f86180c51e93f611e9b5c11c1a8c WatchSource:0}: Error finding container a83257c0942addf97feed1f193bf391fbf35f86180c51e93f611e9b5c11c1a8c: Status 404 returned error can't find the container with id a83257c0942addf97feed1f193bf391fbf35f86180c51e93f611e9b5c11c1a8c Sep 14 01:46:10.831644 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.831620 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.833825 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.833803 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8e5935-serving-cert\"" Sep 14 01:46:10.838591 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.838565 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 01:46:10.849958 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.849934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.850052 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.849974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.850093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.850060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.850147 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.850125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkws\" (UniqueName: \"kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.951453 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.951416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.951453 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.951452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.951726 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.951578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.951726 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.951629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkws\" (UniqueName: \"kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.951726 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:46:10.951718 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-serving-cert: secret "maas-api-e2e-worker-w4-8e5935-serving-cert" not found Sep 14 01:46:10.951878 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:46:10.951786 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls podName:3b94e090-72be-4b9b-9265-0cee35429cd4 nodeName:}" failed. No retries permitted until 2026-09-14 01:46:11.451769658 +0000 UTC m=+918.124885813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls") pod "maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" (UID: "3b94e090-72be-4b9b-9265-0cee35429cd4") : secret "maas-api-e2e-worker-w4-8e5935-serving-cert" not found Sep 14 01:46:10.952101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.952078 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.954107 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.954082 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:10.960600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:10.960580 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkws\" (UniqueName: \"kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:11.455959 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.455922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:11.458558 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.458522 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:11.650998 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.650965 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" event={"ID":"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf","Type":"ContainerStarted","Data":"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c"} Sep 14 01:46:11.651171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.651022 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" event={"ID":"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf","Type":"ContainerStarted","Data":"a83257c0942addf97feed1f193bf391fbf35f86180c51e93f611e9b5c11c1a8c"} Sep 14 01:46:11.651301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.651178 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:11.666755 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.666709 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" podStartSLOduration=2.191719363 podStartE2EDuration="2.666694262s" totalImagesPullingTime="474.974899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:11.665346955 +0000 UTC m=+918.338463129" watchObservedRunningTime="2026-09-14 01:46:11.666694262 +0000 UTC m=+918.339810437" Sep 14 01:46:11.746873 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:11.746842 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:12.024509 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.024483 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 01:46:12.027151 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:12.027116 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b94e090_72be_4b9b_9265_0cee35429cd4.slice/crio-eae239f5c5ef631e47b07fd4ecc71e7c0f2edff1a4db12f965a5f2dbf2031a6a WatchSource:0}: Error finding container eae239f5c5ef631e47b07fd4ecc71e7c0f2edff1a4db12f965a5f2dbf2031a6a: Status 404 returned error can't find the container with id eae239f5c5ef631e47b07fd4ecc71e7c0f2edff1a4db12f965a5f2dbf2031a6a Sep 14 01:46:12.658180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.658132 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" event={"ID":"3b94e090-72be-4b9b-9265-0cee35429cd4","Type":"ContainerStarted","Data":"ded63bde2c0a7545b687a6e3809237b61079a1df333c2c0d4c78df2b50aaf7cf"} Sep 14 01:46:12.658408 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.658196 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" event={"ID":"3b94e090-72be-4b9b-9265-0cee35429cd4","Type":"ContainerStarted","Data":"eae239f5c5ef631e47b07fd4ecc71e7c0f2edff1a4db12f965a5f2dbf2031a6a"} Sep 14 01:46:12.674772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.674726 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" podStartSLOduration=2.341863023 podStartE2EDuration="2.674710024s" totalImagesPullingTime="332.847001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:12.673693785 +0000 UTC m=+919.346809960" watchObservedRunningTime="2026-09-14 01:46:12.674710024 +0000 UTC m=+919.347826199" Sep 14 01:46:12.827725 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.827674 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:46:12.830909 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.830886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.833281 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.833241 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a36bf3-serving-cert\"" Sep 14 01:46:12.841118 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.841084 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:46:12.870630 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.870601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.870784 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.870641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.870784 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.870670 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfg6r\" (UniqueName: \"kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.870784 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.870742 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.972231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.972146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.972231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.972181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.972231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.972216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfg6r\" (UniqueName: \"kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.972527 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.972274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.972906 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.972880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.974848 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.974826 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.974948 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.974872 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:12.980616 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:12.980591 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfg6r\" (UniqueName: \"kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r\") pod \"maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:13.144810 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.144778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:13.426189 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.426154 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:46:13.429339 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:13.429242 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e83eda_6b57_4d54_a01f_0d717807c725.slice/crio-f0e744f2016437c2c0aac07bce68c2137ec4388e658ddaf85273fabb4f24cff3 WatchSource:0}: Error finding container f0e744f2016437c2c0aac07bce68c2137ec4388e658ddaf85273fabb4f24cff3: Status 404 returned error can't find the container with id f0e744f2016437c2c0aac07bce68c2137ec4388e658ddaf85273fabb4f24cff3 Sep 14 01:46:13.589988 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.589956 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:46:13.594566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.594550 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.596997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.596977 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a36bf3\"" Sep 14 01:46:13.596997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.596987 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a36bf3-dockercfg-4nbs7\"" Sep 14 01:46:13.604643 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.604621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:46:13.674594 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.674544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" event={"ID":"97e83eda-6b57-4d54-a01f-0d717807c725","Type":"ContainerStarted","Data":"f0e744f2016437c2c0aac07bce68c2137ec4388e658ddaf85273fabb4f24cff3"} Sep 14 01:46:13.674880 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.674858 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:13.679345 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.679275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94kwg\" (UniqueName: \"kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.679345 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.679341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.780484 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.780451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94kwg\" (UniqueName: \"kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.780595 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.780520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.781368 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.781343 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.788887 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.788861 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94kwg\" (UniqueName: \"kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg\") pod \"payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:13.906330 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:13.906298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:14.196519 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.196475 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:46:14.196877 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:14.196811 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5ded47_51f6_4eef_a8c9_2e7e6efbe718.slice/crio-94f980af43a977d1556d64b67183ba8ee28bb7f14471ef29118bc337e1192361 WatchSource:0}: Error finding container 94f980af43a977d1556d64b67183ba8ee28bb7f14471ef29118bc337e1192361: Status 404 returned error can't find the container with id 94f980af43a977d1556d64b67183ba8ee28bb7f14471ef29118bc337e1192361 Sep 14 01:46:14.682270 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.682215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" event={"ID":"97e83eda-6b57-4d54-a01f-0d717807c725","Type":"ContainerStarted","Data":"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d"} Sep 14 01:46:14.682441 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.682315 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:14.683925 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.683890 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" event={"ID":"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718","Type":"ContainerStarted","Data":"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8"} Sep 14 01:46:14.684077 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.683960 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" event={"ID":"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718","Type":"ContainerStarted","Data":"94f980af43a977d1556d64b67183ba8ee28bb7f14471ef29118bc337e1192361"} Sep 14 01:46:14.684150 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.684130 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:14.699334 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.699288 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" podStartSLOduration=2.35720176 podStartE2EDuration="2.699245971s" totalImagesPullingTime="342.044211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:14.69750625 +0000 UTC m=+921.370622425" watchObservedRunningTime="2026-09-14 01:46:14.699245971 +0000 UTC m=+921.372362145" Sep 14 01:46:14.714793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.714746 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" podStartSLOduration=1.714732584 podStartE2EDuration="1.714732584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:14.713387798 +0000 UTC m=+921.386503973" watchObservedRunningTime="2026-09-14 01:46:14.714732584 +0000 UTC m=+921.387848786" Sep 14 01:46:14.748071 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.748044 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:46:14.751751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.751729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.754346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.754323 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-bc2e37-openshift-default-dockercfg-948kw\"" Sep 14 01:46:14.764483 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.764457 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:46:14.789176 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789122 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789190 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789567 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789358 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.789696 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.790013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.789986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj667\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891031 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.890992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj667\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891045 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891070 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891158 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891542 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891287 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891542 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891334 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891643 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891643 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891609 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891742 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891658 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.891742 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.891723 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.892124 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.892099 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.893918 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.893890 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.894286 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.894244 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.899275 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.899235 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:14.899443 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:14.899426 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj667\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667\") pod \"e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:15.064598 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.064513 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:15.341097 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.341072 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:46:15.342887 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:15.342861 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c635866_97cf_4998_9b0e_4855a798665c.slice/crio-e254d501b6a8845be5658792d1075b06d23626888f8db560e0e04b60856abbe2 WatchSource:0}: Error finding container e254d501b6a8845be5658792d1075b06d23626888f8db560e0e04b60856abbe2: Status 404 returned error can't find the container with id e254d501b6a8845be5658792d1075b06d23626888f8db560e0e04b60856abbe2 Sep 14 01:46:15.345449 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.345413 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:15.345537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.345495 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:15.345576 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.345537 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:15.691707 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.691673 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerStarted","Data":"8086cc71ede178a7f20f6da792bb18adaaefcf9771530e4ffe1116072df56fb3"} Sep 14 01:46:15.691869 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.691728 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerStarted","Data":"e254d501b6a8845be5658792d1075b06d23626888f8db560e0e04b60856abbe2"} Sep 14 01:46:15.709284 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:15.709212 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podStartSLOduration=1.709190972 podStartE2EDuration="1.709190972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:15.707787069 +0000 UTC m=+922.380903244" watchObservedRunningTime="2026-09-14 01:46:15.709190972 +0000 UTC m=+922.382307149" Sep 14 01:46:16.064990 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:16.064910 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:16.070383 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:16.070357 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:16.697713 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:16.697672 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:16.698793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:16.698770 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:17.664896 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:17.664865 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:46:19.689518 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:19.689486 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 01:46:20.698058 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:20.698028 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:46:21.648486 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.648440 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:46:21.651556 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.651539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.653635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.653612 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-bc2e37\"" Sep 14 01:46:21.687764 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.687724 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:46:21.694100 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.694078 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:46:21.697633 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.697616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.699969 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.699949 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-bc2e37-dockercfg-sm4zx\"" Sep 14 01:46:21.705935 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.705914 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:46:21.774787 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.774753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kng4h\" (UniqueName: \"kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.775057 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.775031 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.775239 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.775221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6t22\" (UniqueName: \"kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.775424 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.775408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.876308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.876271 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6t22\" (UniqueName: \"kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.876579 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.876551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.876818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.876798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kng4h\" (UniqueName: \"kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.876969 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.876953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.877975 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.877951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.879102 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.879079 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.886216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.886181 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kng4h\" (UniqueName: \"kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h\") pod \"payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:21.886336 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.886292 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6t22\" (UniqueName: \"kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22\") pod \"payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:21.993877 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:21.993788 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:22.007409 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.007372 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:22.295972 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.295949 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:46:22.297850 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:22.297823 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b468f23_8b8b_47dd_8241_f6dd6218a08a.slice/crio-fe202cc7aecc63ae3fcaf4982ebfa16800077380e56afcbd9e6b17d1bfffa5ae WatchSource:0}: Error finding container fe202cc7aecc63ae3fcaf4982ebfa16800077380e56afcbd9e6b17d1bfffa5ae: Status 404 returned error can't find the container with id fe202cc7aecc63ae3fcaf4982ebfa16800077380e56afcbd9e6b17d1bfffa5ae Sep 14 01:46:22.335337 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.335222 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:46:22.338145 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:22.338122 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef0f212_790a_4905_a325_5ef93336170a.slice/crio-fac5d64fc68edd99f0fbfc6daee5e4c0171c07703b98f591248cc238751d89ae WatchSource:0}: Error finding container fac5d64fc68edd99f0fbfc6daee5e4c0171c07703b98f591248cc238751d89ae: Status 404 returned error can't find the container with id fac5d64fc68edd99f0fbfc6daee5e4c0171c07703b98f591248cc238751d89ae Sep 14 01:46:22.740756 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.740720 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" event={"ID":"2b468f23-8b8b-47dd-8241-f6dd6218a08a","Type":"ContainerStarted","Data":"4ea4c1ef2000a5cb6616db9bf6da7535391d45217687099a7a87c18ec9d2f378"} Sep 14 01:46:22.741209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.740778 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" event={"ID":"2b468f23-8b8b-47dd-8241-f6dd6218a08a","Type":"ContainerStarted","Data":"fe202cc7aecc63ae3fcaf4982ebfa16800077380e56afcbd9e6b17d1bfffa5ae"} Sep 14 01:46:22.741209 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.740811 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:22.742446 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.742420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" event={"ID":"3ef0f212-790a-4905-a325-5ef93336170a","Type":"ContainerStarted","Data":"1bd35b5e89868c65c706dbd33287447fa5666400d3628e4634ff3749bcdba16d"} Sep 14 01:46:22.742550 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.742461 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" event={"ID":"3ef0f212-790a-4905-a325-5ef93336170a","Type":"ContainerStarted","Data":"fac5d64fc68edd99f0fbfc6daee5e4c0171c07703b98f591248cc238751d89ae"} Sep 14 01:46:22.742550 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.742495 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:22.758360 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.758308 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" podStartSLOduration=1.758288174 podStartE2EDuration="1.758288174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:22.754554645 +0000 UTC m=+929.427670821" watchObservedRunningTime="2026-09-14 01:46:22.758288174 +0000 UTC m=+929.431404350" Sep 14 01:46:22.769367 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:22.769310 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" podStartSLOduration=1.769291245 podStartE2EDuration="1.769291245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:22.768228119 +0000 UTC m=+929.441344296" watchObservedRunningTime="2026-09-14 01:46:22.769291245 +0000 UTC m=+929.442407421" Sep 14 01:46:25.694423 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:25.694388 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:46:29.802419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.802247 2573 generic.go:356] "Generic (PLEG): container finished" podID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerID="f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40" exitCode=0 Sep 14 01:46:29.802419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.802303 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerDied","Data":"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40"} Sep 14 01:46:29.802925 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.802890 2573 scope.go:117] "RemoveContainer" containerID="f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40" Sep 14 01:46:29.804365 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.804307 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:29.804459 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.804422 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:29.804503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:29.804459 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:30.812644 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:30.812615 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerStarted","Data":"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f"} Sep 14 01:46:31.549537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:31.549501 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:31.549746 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:31.549622 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 14 01:46:31.549746 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:31.549669 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 14 01:46:32.549486 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:32.549450 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 14 01:46:32.549865 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:32.549503 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 14 01:46:33.549570 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:33.549525 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 14 01:46:33.549986 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:33.549596 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 14 01:46:33.749693 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:33.749662 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:46:33.750039 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:33.750016 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:46:34.549161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:34.549113 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 14 01:46:34.549382 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:34.549181 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 14 01:46:35.549760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:35.549725 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Sep 14 01:46:35.550101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:35.549782 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Sep 14 01:46:36.553341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.553317 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:36.553702 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.553667 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:36.554387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.554362 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:46:36.847578 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.847498 2573 generic.go:356] "Generic (PLEG): container finished" podID="6c635866-97cf-4998-9b0e-4855a798665c" containerID="8086cc71ede178a7f20f6da792bb18adaaefcf9771530e4ffe1116072df56fb3" exitCode=0 Sep 14 01:46:36.847756 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.847569 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerDied","Data":"8086cc71ede178a7f20f6da792bb18adaaefcf9771530e4ffe1116072df56fb3"} Sep 14 01:46:36.848046 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.848030 2573 scope.go:117] "RemoveContainer" containerID="8086cc71ede178a7f20f6da792bb18adaaefcf9771530e4ffe1116072df56fb3" Sep 14 01:46:36.849358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.849302 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:36.849592 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.849389 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:36.849592 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:36.849436 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:46:37.854456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:37.854417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerStarted","Data":"e703ac1b7c810101b7cb2a37a128bbc216a06009ee1c03158bc51e932c4a1dc9"} Sep 14 01:46:38.065294 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:38.065184 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:38.065294 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:38.065240 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:38.065544 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:38.065318 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:39.065271 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:39.065206 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:39.065777 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:39.065299 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:40.065619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.065580 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:40.066108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.065652 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:40.358332 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.354897 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:46:40.378277 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.377433 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:46:40.378277 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.377569 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.380478 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.380452 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-321811\"" Sep 14 01:46:40.389924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.389898 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.390069 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.390040 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bl6\" (UniqueName: \"kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.493116 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.493076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.494328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.494304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bl6\" (UniqueName: \"kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.494563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.494204 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.505368 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.505336 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bl6\" (UniqueName: \"kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6\") pod \"payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:40.696570 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:40.696539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:41.001199 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.001144 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:46:41.002042 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:41.002020 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041e597c_2ea8_4e14_b195_21a039875ce0.slice/crio-64d7299ef6264da908156a40d95cb1708d3ea37b268cb16e9f21bc31a208c0a3 WatchSource:0}: Error finding container 64d7299ef6264da908156a40d95cb1708d3ea37b268cb16e9f21bc31a208c0a3: Status 404 returned error can't find the container with id 64d7299ef6264da908156a40d95cb1708d3ea37b268cb16e9f21bc31a208c0a3 Sep 14 01:46:41.065639 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.065128 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:41.065639 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.065193 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:41.891450 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.891418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" event={"ID":"041e597c-2ea8-4e14-b195-21a039875ce0","Type":"ContainerStarted","Data":"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0"} Sep 14 01:46:41.891669 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.891476 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" event={"ID":"041e597c-2ea8-4e14-b195-21a039875ce0","Type":"ContainerStarted","Data":"64d7299ef6264da908156a40d95cb1708d3ea37b268cb16e9f21bc31a208c0a3"} Sep 14 01:46:41.891669 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.891578 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:41.907143 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:41.907096 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" podStartSLOduration=1.907080012 podStartE2EDuration="1.907080012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:41.905888421 +0000 UTC m=+948.579004596" watchObservedRunningTime="2026-09-14 01:46:41.907080012 +0000 UTC m=+948.580196187" Sep 14 01:46:42.065675 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:42.065626 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:42.066140 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:42.065701 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:43.065024 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:43.064979 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 14 01:46:43.065231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:43.065053 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 14 01:46:44.069711 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:44.069680 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:44.070167 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:44.070081 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:44.070784 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:44.070761 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:46:52.899878 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:52.899844 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:46:54.629776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.629734 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:46:54.633491 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.633471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.635942 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.635918 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b766dd7c-dockercfg-2sqtx\"" Sep 14 01:46:54.636070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.635956 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b766dd7c\"" Sep 14 01:46:54.643688 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.643651 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:46:54.757810 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.757769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtffv\" (UniqueName: \"kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.758115 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.758088 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.859808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.859756 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtffv\" (UniqueName: \"kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.859998 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.859851 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.860804 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.860780 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.867668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.867633 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtffv\" (UniqueName: \"kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv\") pod \"payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:54.946119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:54.946078 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:55.223311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:55.223287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:46:55.225441 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:46:55.225412 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c0c054_b3eb_4ce8_99ad_b90959388623.slice/crio-5b3f93990560038108b936e2ddf02bf76d5c776dce0573de1e75839f115959a7 WatchSource:0}: Error finding container 5b3f93990560038108b936e2ddf02bf76d5c776dce0573de1e75839f115959a7: Status 404 returned error can't find the container with id 5b3f93990560038108b936e2ddf02bf76d5c776dce0573de1e75839f115959a7 Sep 14 01:46:55.968142 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:55.968107 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" event={"ID":"c8c0c054-b3eb-4ce8-99ad-b90959388623","Type":"ContainerStarted","Data":"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3"} Sep 14 01:46:55.968575 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:55.968165 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" event={"ID":"c8c0c054-b3eb-4ce8-99ad-b90959388623","Type":"ContainerStarted","Data":"5b3f93990560038108b936e2ddf02bf76d5c776dce0573de1e75839f115959a7"} Sep 14 01:46:55.968575 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:55.968219 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:46:55.989921 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:46:55.989858 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" podStartSLOduration=1.989843596 podStartE2EDuration="1.989843596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:46:55.98808388 +0000 UTC m=+962.661200051" watchObservedRunningTime="2026-09-14 01:46:55.989843596 +0000 UTC m=+962.662959771" Sep 14 01:47:06.977676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:06.977647 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:47:21.773325 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:21.773290 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:47:21.773771 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:21.773600 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" podUID="c8c0c054-b3eb-4ce8-99ad-b90959388623" containerName="payload-pre-processing" containerID="cri-o://7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3" gracePeriod=30 Sep 14 01:47:22.034402 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.034344 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:47:22.107407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.107375 2573 generic.go:356] "Generic (PLEG): container finished" podID="c8c0c054-b3eb-4ce8-99ad-b90959388623" containerID="7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3" exitCode=0 Sep 14 01:47:22.107618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.107425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" event={"ID":"c8c0c054-b3eb-4ce8-99ad-b90959388623","Type":"ContainerDied","Data":"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3"} Sep 14 01:47:22.107618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.107435 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" Sep 14 01:47:22.107618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.107472 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822" event={"ID":"c8c0c054-b3eb-4ce8-99ad-b90959388623","Type":"ContainerDied","Data":"5b3f93990560038108b936e2ddf02bf76d5c776dce0573de1e75839f115959a7"} Sep 14 01:47:22.107618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.107520 2573 scope.go:117] "RemoveContainer" containerID="7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3" Sep 14 01:47:22.116861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.116842 2573 scope.go:117] "RemoveContainer" containerID="7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3" Sep 14 01:47:22.117162 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:47:22.117135 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3\": container with ID starting with 7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3 not found: ID does not exist" containerID="7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3" Sep 14 01:47:22.117239 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.117173 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3"} err="failed to get container status \"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3\": rpc error: code = NotFound desc = could not find container \"7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3\": container with ID starting with 7d01576d8ee28ed5072e63fba598e12f7acd22fae1f535c8abe347c4707143b3 not found: ID does not exist" Sep 14 01:47:22.135643 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.135616 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtffv\" (UniqueName: \"kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv\") pod \"c8c0c054-b3eb-4ce8-99ad-b90959388623\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " Sep 14 01:47:22.135775 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.135740 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume\") pod \"c8c0c054-b3eb-4ce8-99ad-b90959388623\" (UID: \"c8c0c054-b3eb-4ce8-99ad-b90959388623\") " Sep 14 01:47:22.136055 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.136033 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8c0c054-b3eb-4ce8-99ad-b90959388623" (UID: "c8c0c054-b3eb-4ce8-99ad-b90959388623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:22.137671 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.137649 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv" (OuterVolumeSpecName: "kube-api-access-qtffv") pod "c8c0c054-b3eb-4ce8-99ad-b90959388623" (UID: "c8c0c054-b3eb-4ce8-99ad-b90959388623"). InnerVolumeSpecName "kube-api-access-qtffv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:22.236566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.236537 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtffv\" (UniqueName: \"kubernetes.io/projected/c8c0c054-b3eb-4ce8-99ad-b90959388623-kube-api-access-qtffv\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:47:22.236566 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.236562 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8c0c054-b3eb-4ce8-99ad-b90959388623-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:47:22.431969 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.431933 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:47:22.435032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:22.435003 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b766dd7c-596f67f6fd-6c822"] Sep 14 01:47:23.864231 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:23.864185 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c0c054-b3eb-4ce8-99ad-b90959388623" path="/var/lib/kubelet/pods/c8c0c054-b3eb-4ce8-99ad-b90959388623/volumes" Sep 14 01:47:27.671603 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.671572 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:27.672002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.671974 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c0c054-b3eb-4ce8-99ad-b90959388623" containerName="payload-pre-processing" Sep 14 01:47:27.672002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.672003 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c0c054-b3eb-4ce8-99ad-b90959388623" containerName="payload-pre-processing" Sep 14 01:47:27.672092 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.672083 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c0c054-b3eb-4ce8-99ad-b90959388623" containerName="payload-pre-processing" Sep 14 01:47:27.675464 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.675447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.677847 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.677822 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-91e5cec3\"" Sep 14 01:47:27.683907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.683508 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:27.793099 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.793061 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88j8\" (UniqueName: \"kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.793523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.793480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.894171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.894134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.894556 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.894212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88j8\" (UniqueName: \"kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.894821 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.894800 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.901467 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.901442 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88j8\" (UniqueName: \"kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8\") pod \"payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:27.988885 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:27.988810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:28.263420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:28.263397 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:28.265210 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:47:28.265174 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474f3213_15af_4c30_90b7_6fe377b8aa17.slice/crio-5b7f13f61b75df7f1701f3fc65816d306ccec9356fdbfdec81aa5c043d6dab0d WatchSource:0}: Error finding container 5b7f13f61b75df7f1701f3fc65816d306ccec9356fdbfdec81aa5c043d6dab0d: Status 404 returned error can't find the container with id 5b7f13f61b75df7f1701f3fc65816d306ccec9356fdbfdec81aa5c043d6dab0d Sep 14 01:47:29.151206 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:29.151172 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" event={"ID":"474f3213-15af-4c30-90b7-6fe377b8aa17","Type":"ContainerStarted","Data":"9b87713f476fdad2263cf50d8c5ce4bcae330215d36c4ba6f4548db988cf98e1"} Sep 14 01:47:29.151601 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:29.151221 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" event={"ID":"474f3213-15af-4c30-90b7-6fe377b8aa17","Type":"ContainerStarted","Data":"5b7f13f61b75df7f1701f3fc65816d306ccec9356fdbfdec81aa5c043d6dab0d"} Sep 14 01:47:29.151601 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:29.151308 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:29.168019 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:29.167964 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" podStartSLOduration=2.167942427 podStartE2EDuration="2.167942427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:47:29.165276727 +0000 UTC m=+995.838392902" watchObservedRunningTime="2026-09-14 01:47:29.167942427 +0000 UTC m=+995.841058604" Sep 14 01:47:40.158435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:40.158403 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:57.108657 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.108614 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:57.109163 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.109032 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" podUID="474f3213-15af-4c30-90b7-6fe377b8aa17" containerName="payload-pre-processing" containerID="cri-o://9b87713f476fdad2263cf50d8c5ce4bcae330215d36c4ba6f4548db988cf98e1" gracePeriod=30 Sep 14 01:47:57.303339 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.303300 2573 generic.go:356] "Generic (PLEG): container finished" podID="474f3213-15af-4c30-90b7-6fe377b8aa17" containerID="9b87713f476fdad2263cf50d8c5ce4bcae330215d36c4ba6f4548db988cf98e1" exitCode=0 Sep 14 01:47:57.303496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.303382 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" event={"ID":"474f3213-15af-4c30-90b7-6fe377b8aa17","Type":"ContainerDied","Data":"9b87713f476fdad2263cf50d8c5ce4bcae330215d36c4ba6f4548db988cf98e1"} Sep 14 01:47:57.384965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.384941 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:57.488588 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.488556 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h88j8\" (UniqueName: \"kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8\") pod \"474f3213-15af-4c30-90b7-6fe377b8aa17\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " Sep 14 01:47:57.488751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.488674 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume\") pod \"474f3213-15af-4c30-90b7-6fe377b8aa17\" (UID: \"474f3213-15af-4c30-90b7-6fe377b8aa17\") " Sep 14 01:47:57.489038 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.489003 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "474f3213-15af-4c30-90b7-6fe377b8aa17" (UID: "474f3213-15af-4c30-90b7-6fe377b8aa17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:47:57.490932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.490900 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8" (OuterVolumeSpecName: "kube-api-access-h88j8") pod "474f3213-15af-4c30-90b7-6fe377b8aa17" (UID: "474f3213-15af-4c30-90b7-6fe377b8aa17"). InnerVolumeSpecName "kube-api-access-h88j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:47:57.589755 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.589721 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h88j8\" (UniqueName: \"kubernetes.io/projected/474f3213-15af-4c30-90b7-6fe377b8aa17-kube-api-access-h88j8\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:47:57.589755 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:57.589755 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/474f3213-15af-4c30-90b7-6fe377b8aa17-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:47:58.311418 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:58.311387 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" Sep 14 01:47:58.311418 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:58.311400 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf" event={"ID":"474f3213-15af-4c30-90b7-6fe377b8aa17","Type":"ContainerDied","Data":"5b7f13f61b75df7f1701f3fc65816d306ccec9356fdbfdec81aa5c043d6dab0d"} Sep 14 01:47:58.312125 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:58.311456 2573 scope.go:117] "RemoveContainer" containerID="9b87713f476fdad2263cf50d8c5ce4bcae330215d36c4ba6f4548db988cf98e1" Sep 14 01:47:58.338268 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:58.338222 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:58.344236 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:58.344198 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-91e5cec3-b44cb79c8-zdpwf"] Sep 14 01:47:59.862376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:47:59.862342 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474f3213-15af-4c30-90b7-6fe377b8aa17" path="/var/lib/kubelet/pods/474f3213-15af-4c30-90b7-6fe377b8aa17/volumes" Sep 14 01:48:05.887683 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.887643 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:05.888165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.888046 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474f3213-15af-4c30-90b7-6fe377b8aa17" containerName="payload-pre-processing" Sep 14 01:48:05.888165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.888057 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="474f3213-15af-4c30-90b7-6fe377b8aa17" containerName="payload-pre-processing" Sep 14 01:48:05.888165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.888134 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="474f3213-15af-4c30-90b7-6fe377b8aa17" containerName="payload-pre-processing" Sep 14 01:48:05.892719 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.892700 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:05.895119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.895089 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b351ffdb-dockercfg-2dtlk\"" Sep 14 01:48:05.895223 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.895120 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b351ffdb\"" Sep 14 01:48:05.900863 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.900227 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:05.970951 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.970907 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:05.971115 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:05.970958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7j2f\" (UniqueName: \"kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.071932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.071898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.071932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.071940 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7j2f\" (UniqueName: \"kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.072677 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.072652 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.079846 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.079819 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7j2f\" (UniqueName: \"kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f\") pod \"payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.205719 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.205631 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:06.475768 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:06.475742 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:06.477907 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:48:06.477880 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb0caaa_76d5_437b_91f5_1f575c86d4f2.slice/crio-3b85b6c58265e3713f40a7847b8486cf24d85b991c7a8be761f3fbfa80d1e892 WatchSource:0}: Error finding container 3b85b6c58265e3713f40a7847b8486cf24d85b991c7a8be761f3fbfa80d1e892: Status 404 returned error can't find the container with id 3b85b6c58265e3713f40a7847b8486cf24d85b991c7a8be761f3fbfa80d1e892 Sep 14 01:48:07.363320 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:07.363281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" event={"ID":"0eb0caaa-76d5-437b-91f5-1f575c86d4f2","Type":"ContainerStarted","Data":"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993"} Sep 14 01:48:07.363801 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:07.363350 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" event={"ID":"0eb0caaa-76d5-437b-91f5-1f575c86d4f2","Type":"ContainerStarted","Data":"3b85b6c58265e3713f40a7847b8486cf24d85b991c7a8be761f3fbfa80d1e892"} Sep 14 01:48:07.363801 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:07.363518 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:07.378993 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:07.378947 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" podStartSLOduration=2.378929267 podStartE2EDuration="2.378929267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:48:07.377810223 +0000 UTC m=+1034.050926399" watchObservedRunningTime="2026-09-14 01:48:07.378929267 +0000 UTC m=+1034.052045445" Sep 14 01:48:16.176116 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.176081 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:48:16.176769 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.176382 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" podUID="3ef0f212-790a-4905-a325-5ef93336170a" containerName="payload-processing" containerID="cri-o://1bd35b5e89868c65c706dbd33287447fa5666400d3628e4634ff3749bcdba16d" gracePeriod=30 Sep 14 01:48:16.181990 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.181963 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:48:16.182235 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.182177 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" podUID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" containerName="payload-pre-processing" containerID="cri-o://4ea4c1ef2000a5cb6616db9bf6da7535391d45217687099a7a87c18ec9d2f378" gracePeriod=30 Sep 14 01:48:16.432216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.431709 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" containerID="4ea4c1ef2000a5cb6616db9bf6da7535391d45217687099a7a87c18ec9d2f378" exitCode=0 Sep 14 01:48:16.432216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.431922 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" event={"ID":"2b468f23-8b8b-47dd-8241-f6dd6218a08a","Type":"ContainerDied","Data":"4ea4c1ef2000a5cb6616db9bf6da7535391d45217687099a7a87c18ec9d2f378"} Sep 14 01:48:16.436537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.436502 2573 generic.go:356] "Generic (PLEG): container finished" podID="3ef0f212-790a-4905-a325-5ef93336170a" containerID="1bd35b5e89868c65c706dbd33287447fa5666400d3628e4634ff3749bcdba16d" exitCode=0 Sep 14 01:48:16.436684 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.436552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" event={"ID":"3ef0f212-790a-4905-a325-5ef93336170a","Type":"ContainerDied","Data":"1bd35b5e89868c65c706dbd33287447fa5666400d3628e4634ff3749bcdba16d"} Sep 14 01:48:16.489321 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.489301 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:48:16.492169 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.492151 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:48:16.677119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kng4h\" (UniqueName: \"kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h\") pod \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " Sep 14 01:48:16.677346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677139 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume\") pod \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\" (UID: \"2b468f23-8b8b-47dd-8241-f6dd6218a08a\") " Sep 14 01:48:16.677346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677216 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6t22\" (UniqueName: \"kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22\") pod \"3ef0f212-790a-4905-a325-5ef93336170a\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " Sep 14 01:48:16.677346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume\") pod \"3ef0f212-790a-4905-a325-5ef93336170a\" (UID: \"3ef0f212-790a-4905-a325-5ef93336170a\") " Sep 14 01:48:16.677627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677601 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b468f23-8b8b-47dd-8241-f6dd6218a08a" (UID: "2b468f23-8b8b-47dd-8241-f6dd6218a08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:16.677701 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677663 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ef0f212-790a-4905-a325-5ef93336170a" (UID: "3ef0f212-790a-4905-a325-5ef93336170a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:16.677767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677750 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ef0f212-790a-4905-a325-5ef93336170a-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:16.677812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.677774 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b468f23-8b8b-47dd-8241-f6dd6218a08a-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:16.680014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.679985 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22" (OuterVolumeSpecName: "kube-api-access-v6t22") pod "3ef0f212-790a-4905-a325-5ef93336170a" (UID: "3ef0f212-790a-4905-a325-5ef93336170a"). InnerVolumeSpecName "kube-api-access-v6t22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:16.680127 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.680026 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h" (OuterVolumeSpecName: "kube-api-access-kng4h") pod "2b468f23-8b8b-47dd-8241-f6dd6218a08a" (UID: "2b468f23-8b8b-47dd-8241-f6dd6218a08a"). InnerVolumeSpecName "kube-api-access-kng4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:16.778619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.778577 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kng4h\" (UniqueName: \"kubernetes.io/projected/2b468f23-8b8b-47dd-8241-f6dd6218a08a-kube-api-access-kng4h\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:16.778619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:16.778620 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6t22\" (UniqueName: \"kubernetes.io/projected/3ef0f212-790a-4905-a325-5ef93336170a-kube-api-access-v6t22\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:17.443049 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.443018 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" Sep 14 01:48:17.443049 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.443029 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r" event={"ID":"3ef0f212-790a-4905-a325-5ef93336170a","Type":"ContainerDied","Data":"fac5d64fc68edd99f0fbfc6daee5e4c0171c07703b98f591248cc238751d89ae"} Sep 14 01:48:17.443592 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.443092 2573 scope.go:117] "RemoveContainer" containerID="1bd35b5e89868c65c706dbd33287447fa5666400d3628e4634ff3749bcdba16d" Sep 14 01:48:17.444551 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.444481 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" event={"ID":"2b468f23-8b8b-47dd-8241-f6dd6218a08a","Type":"ContainerDied","Data":"fe202cc7aecc63ae3fcaf4982ebfa16800077380e56afcbd9e6b17d1bfffa5ae"} Sep 14 01:48:17.444551 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.444534 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh" Sep 14 01:48:17.455443 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.455399 2573 scope.go:117] "RemoveContainer" containerID="4ea4c1ef2000a5cb6616db9bf6da7535391d45217687099a7a87c18ec9d2f378" Sep 14 01:48:17.467984 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.467964 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:48:17.476717 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.476694 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-bc2e37-59bc788665-wlz4r"] Sep 14 01:48:17.486092 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.486072 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:48:17.490369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.490346 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-bc2e37-7595887f8b-m5wvh"] Sep 14 01:48:17.863003 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.862971 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" path="/var/lib/kubelet/pods/2b468f23-8b8b-47dd-8241-f6dd6218a08a/volumes" Sep 14 01:48:17.863365 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:17.863351 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef0f212-790a-4905-a325-5ef93336170a" path="/var/lib/kubelet/pods/3ef0f212-790a-4905-a325-5ef93336170a/volumes" Sep 14 01:48:18.373914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:18.373882 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:31.478576 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.478545 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:31.479124 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.478805 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" podUID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" containerName="payload-pre-processing" containerID="cri-o://cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993" gracePeriod=30 Sep 14 01:48:31.766749 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.766718 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:31.825834 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.825803 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume\") pod \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " Sep 14 01:48:31.825997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.825910 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7j2f\" (UniqueName: \"kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f\") pod \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\" (UID: \"0eb0caaa-76d5-437b-91f5-1f575c86d4f2\") " Sep 14 01:48:31.826287 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.826231 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0eb0caaa-76d5-437b-91f5-1f575c86d4f2" (UID: "0eb0caaa-76d5-437b-91f5-1f575c86d4f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:48:31.828226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.828204 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f" (OuterVolumeSpecName: "kube-api-access-t7j2f") pod "0eb0caaa-76d5-437b-91f5-1f575c86d4f2" (UID: "0eb0caaa-76d5-437b-91f5-1f575c86d4f2"). InnerVolumeSpecName "kube-api-access-t7j2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:48:31.926647 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.926622 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7j2f\" (UniqueName: \"kubernetes.io/projected/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-kube-api-access-t7j2f\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:31.926647 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:31.926654 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eb0caaa-76d5-437b-91f5-1f575c86d4f2-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:48:32.530797 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.530764 2573 generic.go:356] "Generic (PLEG): container finished" podID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" containerID="cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993" exitCode=0 Sep 14 01:48:32.531275 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.530808 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" Sep 14 01:48:32.531275 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.530854 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" event={"ID":"0eb0caaa-76d5-437b-91f5-1f575c86d4f2","Type":"ContainerDied","Data":"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993"} Sep 14 01:48:32.531275 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.530917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9" event={"ID":"0eb0caaa-76d5-437b-91f5-1f575c86d4f2","Type":"ContainerDied","Data":"3b85b6c58265e3713f40a7847b8486cf24d85b991c7a8be761f3fbfa80d1e892"} Sep 14 01:48:32.531275 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.530953 2573 scope.go:117] "RemoveContainer" containerID="cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993" Sep 14 01:48:32.540235 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.540217 2573 scope.go:117] "RemoveContainer" containerID="cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993" Sep 14 01:48:32.540542 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:48:32.540527 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993\": container with ID starting with cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993 not found: ID does not exist" containerID="cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993" Sep 14 01:48:32.540596 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.540550 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993"} err="failed to get container status \"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993\": rpc error: code = NotFound desc = could not find container \"cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993\": container with ID starting with cc7666b71cfb149c05c835c0626d02eb407a799384f302324ad1425b89d0a993 not found: ID does not exist" Sep 14 01:48:32.549427 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.549405 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:32.555730 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:32.555710 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-b351ffdb-597755f666-7ffl9"] Sep 14 01:48:33.862464 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:33.862432 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" path="/var/lib/kubelet/pods/0eb0caaa-76d5-437b-91f5-1f575c86d4f2/volumes" Sep 14 01:48:58.545391 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:58.545359 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:48:58.545940 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:58.545903 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" containerID="cri-o://e703ac1b7c810101b7cb2a37a128bbc216a06009ee1c03158bc51e932c4a1dc9" gracePeriod=30 Sep 14 01:48:59.072288 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:59.072227 2573 patch_prober.go:28] interesting pod/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:48:59.072473 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:48:59.072313 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:49:03.692311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.692276 2573 generic.go:356] "Generic (PLEG): container finished" podID="6c635866-97cf-4998-9b0e-4855a798665c" containerID="e703ac1b7c810101b7cb2a37a128bbc216a06009ee1c03158bc51e932c4a1dc9" exitCode=0 Sep 14 01:49:03.692739 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.692283 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerDied","Data":"e703ac1b7c810101b7cb2a37a128bbc216a06009ee1c03158bc51e932c4a1dc9"} Sep 14 01:49:03.692739 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.692370 2573 scope.go:117] "RemoveContainer" containerID="8086cc71ede178a7f20f6da792bb18adaaefcf9771530e4ffe1116072df56fb3" Sep 14 01:49:03.810070 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.809988 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:49:03.945476 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945442 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945488 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945509 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945539 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj667\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945567 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945584 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945617 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.945676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945671 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.946050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945700 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs\") pod \"6c635866-97cf-4998-9b0e-4855a798665c\" (UID: \"6c635866-97cf-4998-9b0e-4855a798665c\") " Sep 14 01:49:03.946050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945789 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:49:03.946050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945807 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:49:03.946050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.945822 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:49:03.946050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946003 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:49:03.946369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946284 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:03.946369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946306 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:03.946369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946310 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:49:03.946369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946322 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:03.946369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.946338 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c635866-97cf-4998-9b0e-4855a798665c-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:03.948058 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.948031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:49:03.948058 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.948047 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:49:03.948586 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.948568 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667" (OuterVolumeSpecName: "kube-api-access-fj667") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "kube-api-access-fj667". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:49:03.948671 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:03.948586 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c635866-97cf-4998-9b0e-4855a798665c" (UID: "6c635866-97cf-4998-9b0e-4855a798665c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:49:04.046861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.046822 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:04.046861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.046855 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c635866-97cf-4998-9b0e-4855a798665c-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:04.046861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.046866 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj667\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-kube-api-access-fj667\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:04.047126 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.046876 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c635866-97cf-4998-9b0e-4855a798665c-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:04.047126 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.046885 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c635866-97cf-4998-9b0e-4855a798665c-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:49:04.700237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.700201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" event={"ID":"6c635866-97cf-4998-9b0e-4855a798665c","Type":"ContainerDied","Data":"e254d501b6a8845be5658792d1075b06d23626888f8db560e0e04b60856abbe2"} Sep 14 01:49:04.700237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.700226 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq" Sep 14 01:49:04.700745 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.700339 2573 scope.go:117] "RemoveContainer" containerID="e703ac1b7c810101b7cb2a37a128bbc216a06009ee1c03158bc51e932c4a1dc9" Sep 14 01:49:04.724786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.724756 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:49:04.728464 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:04.728439 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-bc2e37-openshift-default-7d776c56fd-f4wmq"] Sep 14 01:49:05.862988 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.862952 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c635866-97cf-4998-9b0e-4855a798665c" path="/var/lib/kubelet/pods/6c635866-97cf-4998-9b0e-4855a798665c/volumes" Sep 14 01:49:05.863366 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863224 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:49:05.863604 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863591 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" containerName="payload-pre-processing" Sep 14 01:49:05.863641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863606 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" containerName="payload-pre-processing" Sep 14 01:49:05.863641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863617 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.863641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863622 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.863641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863632 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" containerName="payload-pre-processing" Sep 14 01:49:05.863641 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863638 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" containerName="payload-pre-processing" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863648 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef0f212-790a-4905-a325-5ef93336170a" containerName="payload-processing" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863654 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef0f212-790a-4905-a325-5ef93336170a" containerName="payload-processing" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863707 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863715 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b468f23-8b8b-47dd-8241-f6dd6218a08a" containerName="payload-pre-processing" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863722 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb0caaa-76d5-437b-91f5-1f575c86d4f2" containerName="payload-pre-processing" Sep 14 01:49:05.863796 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.863728 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef0f212-790a-4905-a325-5ef93336170a" containerName="payload-processing" Sep 14 01:49:05.868082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.868067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:05.870332 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.870312 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7aab97\"" Sep 14 01:49:05.870455 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.870417 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7aab97-dockercfg-4jkjz\"" Sep 14 01:49:05.877451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.877426 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:49:05.965620 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.965586 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpgw\" (UniqueName: \"kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:05.965620 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.965623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:05.984741 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.984715 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:49:05.985183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.985160 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.985183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.985185 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.985376 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.985308 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c635866-97cf-4998-9b0e-4855a798665c" containerName="istio-proxy" Sep 14 01:49:05.988487 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.988473 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:05.998577 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:05.998549 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:49:06.067274 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.067222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpgw\" (UniqueName: \"kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.067460 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.067283 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.067460 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.067351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4gj\" (UniqueName: \"kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.067460 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.067400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.068085 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.068061 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.074966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.074944 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpgw\" (UniqueName: \"kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw\") pod \"payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.168052 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.168014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.168214 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.168129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4gj\" (UniqueName: \"kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.168802 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.168782 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.175422 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.175401 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4gj\" (UniqueName: \"kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj\") pod \"payload-processing-e2e-explicit-7aab97-99945f649-7bmqx\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.180369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.180350 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.300477 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.300445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.453344 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.453320 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:49:06.454766 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:49:06.454735 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9ac5a8_77a5_4aa5_b33e_caf5220d0204.slice/crio-c0a721593212c4c50ec7f1860e7bc08fdbbcbe8bf7f1c27637597eac3f06983f WatchSource:0}: Error finding container c0a721593212c4c50ec7f1860e7bc08fdbbcbe8bf7f1c27637597eac3f06983f: Status 404 returned error can't find the container with id c0a721593212c4c50ec7f1860e7bc08fdbbcbe8bf7f1c27637597eac3f06983f Sep 14 01:49:06.574841 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.574811 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:49:06.576519 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:49:06.576491 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d58ddc6_e2a1_475f_af03_64c41742c5d3.slice/crio-0947fcf91e5f4fb252c601d871d168cd72b34ff4ddcb5127ae1ae6263379eeca WatchSource:0}: Error finding container 0947fcf91e5f4fb252c601d871d168cd72b34ff4ddcb5127ae1ae6263379eeca: Status 404 returned error can't find the container with id 0947fcf91e5f4fb252c601d871d168cd72b34ff4ddcb5127ae1ae6263379eeca Sep 14 01:49:06.716047 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.715949 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" event={"ID":"1d58ddc6-e2a1-475f-af03-64c41742c5d3","Type":"ContainerStarted","Data":"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce"} Sep 14 01:49:06.716047 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.716018 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:49:06.716047 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.716043 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" event={"ID":"1d58ddc6-e2a1-475f-af03-64c41742c5d3","Type":"ContainerStarted","Data":"0947fcf91e5f4fb252c601d871d168cd72b34ff4ddcb5127ae1ae6263379eeca"} Sep 14 01:49:06.717735 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.717707 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" event={"ID":"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204","Type":"ContainerStarted","Data":"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb"} Sep 14 01:49:06.717834 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.717759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" event={"ID":"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204","Type":"ContainerStarted","Data":"c0a721593212c4c50ec7f1860e7bc08fdbbcbe8bf7f1c27637597eac3f06983f"} Sep 14 01:49:06.717879 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.717831 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:06.731668 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.731616 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" podStartSLOduration=1.731597255 podStartE2EDuration="1.731597255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:49:06.729736034 +0000 UTC m=+1093.402852209" watchObservedRunningTime="2026-09-14 01:49:06.731597255 +0000 UTC m=+1093.404713432" Sep 14 01:49:06.744628 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:06.744315 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" podStartSLOduration=1.744298475 podStartE2EDuration="1.744298475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:49:06.744040727 +0000 UTC m=+1093.417156905" watchObservedRunningTime="2026-09-14 01:49:06.744298475 +0000 UTC m=+1093.417414651" Sep 14 01:49:17.724842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:17.724814 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:49:17.725335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:49:17.725068 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:50:05.594989 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.594945 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:50:05.600501 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.600472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.603087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.603062 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-287e139e-openshift-default-dockercfg-d8l8c\"" Sep 14 01:50:05.609484 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.609461 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:50:05.735009 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.734970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735125 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735190 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmgn\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735356 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735356 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735423 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.735423 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.735411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836479 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836690 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836690 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836690 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836875 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmgn\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836875 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836783 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836875 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.836875 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836849 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837037 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836910 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837037 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836945 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837037 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.836999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837164 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.837037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837318 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.837294 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.837387 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.837317 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.839196 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.839164 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.839498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.839481 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.848830 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.848774 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.848932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.848909 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmgn\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn\") pod \"e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:05.912983 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:05.912945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:06.230287 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:06.230264 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:50:06.232455 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:50:06.232429 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f491eeb_475e_4510_b81f_62dcc67e6866.slice/crio-d321c1e5909d5fcf2fc16744246c29f064bd54781df6845209c2425359354562 WatchSource:0}: Error finding container d321c1e5909d5fcf2fc16744246c29f064bd54781df6845209c2425359354562: Status 404 returned error can't find the container with id d321c1e5909d5fcf2fc16744246c29f064bd54781df6845209c2425359354562 Sep 14 01:50:06.234754 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:06.234720 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:06.234855 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:06.234797 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:06.234855 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:06.234844 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:07.017699 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:07.017667 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerStarted","Data":"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b"} Sep 14 01:50:07.018085 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:07.017730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerStarted","Data":"d321c1e5909d5fcf2fc16744246c29f064bd54781df6845209c2425359354562"} Sep 14 01:50:07.039415 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:07.039369 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podStartSLOduration=2.039350985 podStartE2EDuration="2.039350985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:50:07.037809849 +0000 UTC m=+1153.710926028" watchObservedRunningTime="2026-09-14 01:50:07.039350985 +0000 UTC m=+1153.712467159" Sep 14 01:50:07.913899 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:07.913863 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:07.919082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:07.919057 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:08.022741 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:08.022712 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:08.023760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:08.023741 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:14.992457 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:14.992378 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:14.996303 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:14.996285 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:14.998564 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:14.998543 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-287e139e\"" Sep 14 01:50:14.999358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:14.999074 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-287e139e-dockercfg-r5dbw\"" Sep 14 01:50:15.008502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.007962 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:15.015404 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.015381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.015548 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.015522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqltc\" (UniqueName: \"kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.116638 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.116598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqltc\" (UniqueName: \"kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.116839 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.116688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.117517 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.117492 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.124045 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.124021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqltc\" (UniqueName: \"kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc\") pod \"payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.313437 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.313347 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:15.583899 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:15.583873 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:15.586846 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:50:15.586819 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf298f139_8fac_488e_a766_b7fbbef7354b.slice/crio-10bd15b7c4a00e5fd9cbb5ee1a9c5be2ecd2509026316824ec4275ea859fea9b WatchSource:0}: Error finding container 10bd15b7c4a00e5fd9cbb5ee1a9c5be2ecd2509026316824ec4275ea859fea9b: Status 404 returned error can't find the container with id 10bd15b7c4a00e5fd9cbb5ee1a9c5be2ecd2509026316824ec4275ea859fea9b Sep 14 01:50:16.067573 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:16.067538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" event={"ID":"f298f139-8fac-488e-a766-b7fbbef7354b","Type":"ContainerStarted","Data":"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c"} Sep 14 01:50:16.067957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:16.067597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" event={"ID":"f298f139-8fac-488e-a766-b7fbbef7354b","Type":"ContainerStarted","Data":"10bd15b7c4a00e5fd9cbb5ee1a9c5be2ecd2509026316824ec4275ea859fea9b"} Sep 14 01:50:16.067957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:16.067731 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:16.083444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:16.083285 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" podStartSLOduration=2.083247335 podStartE2EDuration="2.083247335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:50:16.082710097 +0000 UTC m=+1162.755826275" watchObservedRunningTime="2026-09-14 01:50:16.083247335 +0000 UTC m=+1162.756363510" Sep 14 01:50:27.076456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:27.076426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:38.024246 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.024149 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:38.024246 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.024201 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:38.197844 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.197810 2573 generic.go:356] "Generic (PLEG): container finished" podID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerID="2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b" exitCode=0 Sep 14 01:50:38.198035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.197867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerDied","Data":"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b"} Sep 14 01:50:38.198367 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.198337 2573 scope.go:117] "RemoveContainer" containerID="2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b" Sep 14 01:50:38.199684 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.199652 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:38.199788 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.199741 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:38.199851 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:38.199791 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:50:39.205182 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:39.205147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerStarted","Data":"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1"} Sep 14 01:50:39.914226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:39.914184 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:39.914444 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:39.914423 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:39.914489 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:39.914472 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:40.913506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:40.913468 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:40.913907 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:40.913527 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:41.913525 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:41.913493 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:41.913990 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:41.913549 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:42.913555 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:42.913512 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:42.913933 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:42.913575 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:43.913778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:43.913740 2573 patch_prober.go:28] interesting pod/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Sep 14 01:50:43.914307 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:43.913807 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Sep 14 01:50:44.918072 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:44.918040 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:44.918545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:44.918428 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:44.919160 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:44.919142 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:50:58.472112 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.472074 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:58.473000 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.472942 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" podUID="f298f139-8fac-488e-a766-b7fbbef7354b" containerName="payload-pre-processing" containerID="cri-o://e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c" gracePeriod=30 Sep 14 01:50:58.739296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.739198 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:58.875828 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.875784 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqltc\" (UniqueName: \"kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc\") pod \"f298f139-8fac-488e-a766-b7fbbef7354b\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " Sep 14 01:50:58.876019 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.875905 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume\") pod \"f298f139-8fac-488e-a766-b7fbbef7354b\" (UID: \"f298f139-8fac-488e-a766-b7fbbef7354b\") " Sep 14 01:50:58.876294 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.876242 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f298f139-8fac-488e-a766-b7fbbef7354b" (UID: "f298f139-8fac-488e-a766-b7fbbef7354b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:50:58.878504 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.878477 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc" (OuterVolumeSpecName: "kube-api-access-pqltc") pod "f298f139-8fac-488e-a766-b7fbbef7354b" (UID: "f298f139-8fac-488e-a766-b7fbbef7354b"). InnerVolumeSpecName "kube-api-access-pqltc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:50:58.977027 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.976988 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f298f139-8fac-488e-a766-b7fbbef7354b-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:50:58.977027 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:58.977022 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqltc\" (UniqueName: \"kubernetes.io/projected/f298f139-8fac-488e-a766-b7fbbef7354b-kube-api-access-pqltc\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:50:59.312348 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.312313 2573 generic.go:356] "Generic (PLEG): container finished" podID="f298f139-8fac-488e-a766-b7fbbef7354b" containerID="e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c" exitCode=0 Sep 14 01:50:59.312523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.312356 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" Sep 14 01:50:59.312523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.312401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" event={"ID":"f298f139-8fac-488e-a766-b7fbbef7354b","Type":"ContainerDied","Data":"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c"} Sep 14 01:50:59.312523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.312455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5" event={"ID":"f298f139-8fac-488e-a766-b7fbbef7354b","Type":"ContainerDied","Data":"10bd15b7c4a00e5fd9cbb5ee1a9c5be2ecd2509026316824ec4275ea859fea9b"} Sep 14 01:50:59.312523 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.312476 2573 scope.go:117] "RemoveContainer" containerID="e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c" Sep 14 01:50:59.322477 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.322462 2573 scope.go:117] "RemoveContainer" containerID="e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c" Sep 14 01:50:59.322742 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:50:59.322723 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c\": container with ID starting with e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c not found: ID does not exist" containerID="e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c" Sep 14 01:50:59.322812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.322749 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c"} err="failed to get container status \"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c\": rpc error: code = NotFound desc = could not find container \"e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c\": container with ID starting with e7216ee1ad04541a9721318bdaa5ab0af9a28f965c4e7e0a73b114d69a1b535c not found: ID does not exist" Sep 14 01:50:59.335776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.335749 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:59.339653 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.339631 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-287e139e-58f79d8b78-sz9g5"] Sep 14 01:50:59.862420 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:50:59.862387 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f298f139-8fac-488e-a766-b7fbbef7354b" path="/var/lib/kubelet/pods/f298f139-8fac-488e-a766-b7fbbef7354b/volumes" Sep 14 01:51:04.964389 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:04.964348 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:51:04.964884 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:04.964575 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" containerID="cri-o://8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1" gracePeriod=30 Sep 14 01:51:10.224013 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.223986 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:51:10.370989 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.370900 2573 generic.go:356] "Generic (PLEG): container finished" podID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerID="8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1" exitCode=0 Sep 14 01:51:10.370989 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.370953 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" Sep 14 01:51:10.371207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.370990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerDied","Data":"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1"} Sep 14 01:51:10.371207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.371039 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt" event={"ID":"3f491eeb-475e-4510-b81f-62dcc67e6866","Type":"ContainerDied","Data":"d321c1e5909d5fcf2fc16744246c29f064bd54781df6845209c2425359354562"} Sep 14 01:51:10.371207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.371061 2573 scope.go:117] "RemoveContainer" containerID="8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1" Sep 14 01:51:10.380551 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.380532 2573 scope.go:117] "RemoveContainer" containerID="2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b" Sep 14 01:51:10.388516 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.388498 2573 scope.go:117] "RemoveContainer" containerID="8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1" Sep 14 01:51:10.388800 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:51:10.388781 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1\": container with ID starting with 8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1 not found: ID does not exist" containerID="8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1" Sep 14 01:51:10.388847 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.388810 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1"} err="failed to get container status \"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1\": rpc error: code = NotFound desc = could not find container \"8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1\": container with ID starting with 8b2d377753f8eeeb747d44f780cc06f0215db7fd47499d398b3fae6f58e258d1 not found: ID does not exist" Sep 14 01:51:10.388847 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.388830 2573 scope.go:117] "RemoveContainer" containerID="2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b" Sep 14 01:51:10.389105 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:51:10.389085 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b\": container with ID starting with 2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b not found: ID does not exist" containerID="2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b" Sep 14 01:51:10.389152 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.389112 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b"} err="failed to get container status \"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b\": rpc error: code = NotFound desc = could not find container \"2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b\": container with ID starting with 2ca2c8106acb29cc61faa02e1e2c7e4d5349ffd41b1f750d4750cce45fc4144b not found: ID does not exist" Sep 14 01:51:10.396479 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396459 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396541 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396492 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396541 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396513 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396625 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396549 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396625 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396606 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndmgn\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396705 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396633 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396758 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396701 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396758 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396738 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396758 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396747 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:51:10.396908 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396790 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy\") pod \"3f491eeb-475e-4510-b81f-62dcc67e6866\" (UID: \"3f491eeb-475e-4510-b81f-62dcc67e6866\") " Sep 14 01:51:10.396908 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396862 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:51:10.397016 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.396983 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:51:10.397016 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397000 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:51:10.397127 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397082 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:51:10.397185 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397141 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.397185 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397160 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.397185 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397178 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.397337 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.397200 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.398956 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.398924 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn" (OuterVolumeSpecName: "kube-api-access-ndmgn") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "kube-api-access-ndmgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:51:10.399052 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.398978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:51:10.399530 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.399509 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:51:10.399643 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.399545 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f491eeb-475e-4510-b81f-62dcc67e6866" (UID: "3f491eeb-475e-4510-b81f-62dcc67e6866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:51:10.497981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.497942 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmgn\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-kube-api-access-ndmgn\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.497981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.497972 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f491eeb-475e-4510-b81f-62dcc67e6866-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.497981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.497982 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.497981 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.497990 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.498282 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.497998 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f491eeb-475e-4510-b81f-62dcc67e6866-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:10.693046 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.693014 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:51:10.695270 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:10.695232 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-287e139e-openshift-default-5549b99b84-vlhvt"] Sep 14 01:51:11.863505 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:11.863472 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" path="/var/lib/kubelet/pods/3f491eeb-475e-4510-b81f-62dcc67e6866/volumes" Sep 14 01:51:18.191613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.191584 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:51:18.192195 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192178 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:18.192243 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192198 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:18.192243 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192227 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f298f139-8fac-488e-a766-b7fbbef7354b" containerName="payload-pre-processing" Sep 14 01:51:18.192243 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192233 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f298f139-8fac-488e-a766-b7fbbef7354b" containerName="payload-pre-processing" Sep 14 01:51:18.192369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192335 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f298f139-8fac-488e-a766-b7fbbef7354b" containerName="payload-pre-processing" Sep 14 01:51:18.192369 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.192344 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:18.197536 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.197516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.200032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.200009 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-38e192-openshift-default-dockercfg-ft6tl\"" Sep 14 01:51:18.204233 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.204212 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:51:18.288517 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288697 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288697 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288584 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288697 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288707 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288905 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288905 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2vg\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.288905 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.288883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389591 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389838 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389873 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389859 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389876 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.389924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390021 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.389950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390021 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390107 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2vg\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390288 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390680 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390323 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390683 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.390778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.390423 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.392334 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.392313 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.392421 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.392393 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.397528 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.397510 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.397934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.397912 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2vg\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg\") pod \"e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.509807 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.509727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:18.788534 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.788507 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:51:18.789828 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:51:18.789802 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc217e024_389a_4e96_a054_2edf4df559f7.slice/crio-2c9d1e09a8d9b8981defc9ba4342e54963358425ebe4fcc66aac6d450d63bfc8 WatchSource:0}: Error finding container 2c9d1e09a8d9b8981defc9ba4342e54963358425ebe4fcc66aac6d450d63bfc8: Status 404 returned error can't find the container with id 2c9d1e09a8d9b8981defc9ba4342e54963358425ebe4fcc66aac6d450d63bfc8 Sep 14 01:51:18.792155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.792124 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:18.792241 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.792186 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:18.792241 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:18.792218 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:19.419216 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:19.419186 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerStarted","Data":"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13"} Sep 14 01:51:19.419587 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:19.419234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerStarted","Data":"2c9d1e09a8d9b8981defc9ba4342e54963358425ebe4fcc66aac6d450d63bfc8"} Sep 14 01:51:19.439630 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:19.439586 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podStartSLOduration=1.4395653400000001 podStartE2EDuration="1.43956534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:19.436591442 +0000 UTC m=+1226.109707619" watchObservedRunningTime="2026-09-14 01:51:19.43956534 +0000 UTC m=+1226.112681514" Sep 14 01:51:19.509960 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:19.509932 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:19.515322 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:19.515298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:20.424737 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:20.424698 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:20.425848 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:20.425824 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:25.475728 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.475697 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:51:25.476126 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.476112 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:25.476174 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.476127 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:25.476210 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.476195 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f491eeb-475e-4510-b81f-62dcc67e6866" containerName="istio-proxy" Sep 14 01:51:25.479222 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.479205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.481843 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.481819 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-38e192-dockercfg-rc89l\"" Sep 14 01:51:25.482195 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.482177 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-38e192\"" Sep 14 01:51:25.491976 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.490534 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:51:25.563246 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.563200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.563479 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.563460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.664655 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.664614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.664850 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.664676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.665416 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.665389 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.672578 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.672542 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp\") pod \"payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:25.792662 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:25.792579 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:26.064082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:26.064049 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:51:26.065651 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:51:26.065611 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487df5c2_8f99_4197_a2a5_372291012fae.slice/crio-bc4a898558f3f4c27c84378b36044906548a010f35872d09be552c62122eafa1 WatchSource:0}: Error finding container bc4a898558f3f4c27c84378b36044906548a010f35872d09be552c62122eafa1: Status 404 returned error can't find the container with id bc4a898558f3f4c27c84378b36044906548a010f35872d09be552c62122eafa1 Sep 14 01:51:26.455429 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:26.455389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" event={"ID":"487df5c2-8f99-4197-a2a5-372291012fae","Type":"ContainerStarted","Data":"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259"} Sep 14 01:51:26.455629 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:26.455453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" event={"ID":"487df5c2-8f99-4197-a2a5-372291012fae","Type":"ContainerStarted","Data":"bc4a898558f3f4c27c84378b36044906548a010f35872d09be552c62122eafa1"} Sep 14 01:51:26.455629 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:26.455606 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:26.471142 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:26.471092 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" podStartSLOduration=1.471075424 podStartE2EDuration="1.471075424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:26.470021285 +0000 UTC m=+1233.143137462" watchObservedRunningTime="2026-09-14 01:51:26.471075424 +0000 UTC m=+1233.144191599" Sep 14 01:51:37.464394 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:37.464358 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:51:40.881395 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:40.880235 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:51:40.882115 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:40.882080 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" podUID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" containerName="payload-processing" containerID="cri-o://b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce" gracePeriod=30 Sep 14 01:51:40.895297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:40.894138 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:51:40.895465 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:40.895412 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" podUID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" containerName="payload-pre-processing" containerID="cri-o://0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb" gracePeriod=30 Sep 14 01:51:41.223050 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.222973 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:51:41.244877 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.244850 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:51:41.324605 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.324563 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume\") pod \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " Sep 14 01:51:41.324769 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.324691 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4gj\" (UniqueName: \"kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj\") pod \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\" (UID: \"1d58ddc6-e2a1-475f-af03-64c41742c5d3\") " Sep 14 01:51:41.325506 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.325477 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d58ddc6-e2a1-475f-af03-64c41742c5d3" (UID: "1d58ddc6-e2a1-475f-af03-64c41742c5d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:51:41.327049 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.327029 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj" (OuterVolumeSpecName: "kube-api-access-tn4gj") pod "1d58ddc6-e2a1-475f-af03-64c41742c5d3" (UID: "1d58ddc6-e2a1-475f-af03-64c41742c5d3"). InnerVolumeSpecName "kube-api-access-tn4gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:51:41.425843 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.425751 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume\") pod \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " Sep 14 01:51:41.426030 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.425900 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqpgw\" (UniqueName: \"kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw\") pod \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\" (UID: \"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204\") " Sep 14 01:51:41.426106 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.426093 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d58ddc6-e2a1-475f-af03-64c41742c5d3-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:41.426106 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.426103 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4gj\" (UniqueName: \"kubernetes.io/projected/1d58ddc6-e2a1-475f-af03-64c41742c5d3-kube-api-access-tn4gj\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:41.426215 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.426096 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" (UID: "7a9ac5a8-77a5-4aa5-b33e-caf5220d0204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:51:41.428009 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.427987 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw" (OuterVolumeSpecName: "kube-api-access-dqpgw") pod "7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" (UID: "7a9ac5a8-77a5-4aa5-b33e-caf5220d0204"). InnerVolumeSpecName "kube-api-access-dqpgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:51:41.527311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.527199 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqpgw\" (UniqueName: \"kubernetes.io/projected/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-kube-api-access-dqpgw\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:41.527311 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.527227 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:51:41.540698 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.540671 2573 generic.go:356] "Generic (PLEG): container finished" podID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" containerID="b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce" exitCode=0 Sep 14 01:51:41.540870 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.540719 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" Sep 14 01:51:41.540870 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.540752 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" event={"ID":"1d58ddc6-e2a1-475f-af03-64c41742c5d3","Type":"ContainerDied","Data":"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce"} Sep 14 01:51:41.540870 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.540791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx" event={"ID":"1d58ddc6-e2a1-475f-af03-64c41742c5d3","Type":"ContainerDied","Data":"0947fcf91e5f4fb252c601d871d168cd72b34ff4ddcb5127ae1ae6263379eeca"} Sep 14 01:51:41.540870 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.540811 2573 scope.go:117] "RemoveContainer" containerID="b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce" Sep 14 01:51:41.542186 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.542161 2573 generic.go:356] "Generic (PLEG): container finished" podID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" containerID="0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb" exitCode=0 Sep 14 01:51:41.542321 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.542193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" event={"ID":"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204","Type":"ContainerDied","Data":"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb"} Sep 14 01:51:41.542321 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.542209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" Sep 14 01:51:41.542321 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.542288 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p" event={"ID":"7a9ac5a8-77a5-4aa5-b33e-caf5220d0204","Type":"ContainerDied","Data":"c0a721593212c4c50ec7f1860e7bc08fdbbcbe8bf7f1c27637597eac3f06983f"} Sep 14 01:51:41.552585 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.552568 2573 scope.go:117] "RemoveContainer" containerID="b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce" Sep 14 01:51:41.552925 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:51:41.552903 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce\": container with ID starting with b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce not found: ID does not exist" containerID="b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce" Sep 14 01:51:41.552997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.552933 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce"} err="failed to get container status \"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce\": rpc error: code = NotFound desc = could not find container \"b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce\": container with ID starting with b974d7903860f8db7a4575f48bf521833bf7dd87d795aeb7187553f9f61090ce not found: ID does not exist" Sep 14 01:51:41.552997 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.552949 2573 scope.go:117] "RemoveContainer" containerID="0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb" Sep 14 01:51:41.570654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.568395 2573 scope.go:117] "RemoveContainer" containerID="0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb" Sep 14 01:51:41.570654 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:51:41.568730 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb\": container with ID starting with 0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb not found: ID does not exist" containerID="0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb" Sep 14 01:51:41.570654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.568765 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb"} err="failed to get container status \"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb\": rpc error: code = NotFound desc = could not find container \"0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb\": container with ID starting with 0a3388fddadee0621070ab51f38ea665bdd3b2ecd7db3be3743e32ae9c9840eb not found: ID does not exist" Sep 14 01:51:41.571362 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.571328 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:51:41.579392 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.579356 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7aab97-99945f649-7bmqx"] Sep 14 01:51:41.593536 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.593506 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:51:41.598937 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.598914 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7aab97-5496776878-pz99p"] Sep 14 01:51:41.867094 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.866997 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" path="/var/lib/kubelet/pods/1d58ddc6-e2a1-475f-af03-64c41742c5d3/volumes" Sep 14 01:51:41.867614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:41.867588 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" path="/var/lib/kubelet/pods/7a9ac5a8-77a5-4aa5-b33e-caf5220d0204/volumes" Sep 14 01:51:47.000492 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.000458 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:51:47.001157 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001138 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" containerName="payload-processing" Sep 14 01:51:47.001226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001163 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" containerName="payload-processing" Sep 14 01:51:47.001226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001184 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" containerName="payload-pre-processing" Sep 14 01:51:47.001226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001193 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" containerName="payload-pre-processing" Sep 14 01:51:47.001354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001336 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d58ddc6-e2a1-475f-af03-64c41742c5d3" containerName="payload-processing" Sep 14 01:51:47.001398 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.001360 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a9ac5a8-77a5-4aa5-b33e-caf5220d0204" containerName="payload-pre-processing" Sep 14 01:51:47.006452 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.006427 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.008914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.008892 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7ff93e-openshift-default-dockercfg-m5xjm\"" Sep 14 01:51:47.015194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.015172 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:51:47.187032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.186985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187055 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dkb\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187351 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.187701 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.187396 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.288893 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.288803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.288893 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.288842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.288893 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.288889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.288951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dkb\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.288999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289307 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289456 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289439 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289676 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289630 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.289736 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289724 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.290018 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.289995 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.291955 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.291922 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.292042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.292024 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.296459 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.296432 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.296819 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.296795 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dkb\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb\") pod \"e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.320589 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.320564 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:47.580007 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.579930 2573 generic.go:356] "Generic (PLEG): container finished" podID="c217e024-389a-4e96-a054-2edf4df559f7" containerID="089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13" exitCode=0 Sep 14 01:51:47.580204 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.580004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerDied","Data":"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13"} Sep 14 01:51:47.580541 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.580524 2573 scope.go:117] "RemoveContainer" containerID="089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13" Sep 14 01:51:47.581873 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.581841 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:47.581966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.581917 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:47.581966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.581951 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:47.602561 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.602535 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:51:47.605102 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:51:47.604975 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72585742_6b07_4546_9a88_b4da4c187214.slice/crio-6702433ef18ae305524e3c8427e9440af4d7aa5a33a9bebcddaf5d8aef8389f4 WatchSource:0}: Error finding container 6702433ef18ae305524e3c8427e9440af4d7aa5a33a9bebcddaf5d8aef8389f4: Status 404 returned error can't find the container with id 6702433ef18ae305524e3c8427e9440af4d7aa5a33a9bebcddaf5d8aef8389f4 Sep 14 01:51:47.607172 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.607140 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:47.607296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.607213 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:47.607296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:47.607277 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:51:48.588425 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:48.588389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerStarted","Data":"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541"} Sep 14 01:51:48.590063 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:48.590037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerStarted","Data":"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0"} Sep 14 01:51:48.590188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:48.590081 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerStarted","Data":"6702433ef18ae305524e3c8427e9440af4d7aa5a33a9bebcddaf5d8aef8389f4"} Sep 14 01:51:48.622843 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:48.622798 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podStartSLOduration=2.622784002 podStartE2EDuration="2.622784002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:48.621722856 +0000 UTC m=+1255.294839030" watchObservedRunningTime="2026-09-14 01:51:48.622784002 +0000 UTC m=+1255.295900174" Sep 14 01:51:49.321418 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.321383 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:49.343823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.343788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:49.510294 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.510232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:49.511560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.511532 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:51:49.511708 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.511585 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 14 01:51:49.595285 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.595161 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:49.596284 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:49.596239 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:51:50.425549 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.425518 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:50.510618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.510585 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:51:50.510813 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.510645 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 14 01:51:50.721786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.721712 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:51:50.725834 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.725813 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.729048 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.729015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-99c616\"/\"default-dockercfg-pst88\"" Sep 14 01:51:50.729048 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.729015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-99c616\"/\"distinct-w0-99c616-kserve-self-signed-certs\"" Sep 14 01:51:50.729226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.729074 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-99c616\"/\"kube-root-ca.crt\"" Sep 14 01:51:50.729226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.729078 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-99c616\"/\"openshift-service-ca.crt\"" Sep 14 01:51:50.736386 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.736364 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:51:50.826087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826056 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.826087 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826094 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.826341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8q9d\" (UniqueName: \"kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.826341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826144 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.826341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.826341 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.826318 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.926942 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.926902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927135 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.926961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927135 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.926988 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8q9d\" (UniqueName: \"kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927135 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927135 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927449 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927449 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927413 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927449 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927441 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.927608 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.927535 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.929767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.929737 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.930002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.929982 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:50.935167 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:50.935145 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8q9d\" (UniqueName: \"kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d\") pod \"distinct-w0-99c616-kserve-d8849655f-c5sf9\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:51.037717 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.037619 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:51:51.334751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.334723 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:51:51.336693 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:51:51.336649 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0465dd4d_0dfe_4768_96ca_1a76405f5f90.slice/crio-76b8281a2324ae2d7f5ccbe636914e426b411b997208a9ac52dd9d103b2538c2 WatchSource:0}: Error finding container 76b8281a2324ae2d7f5ccbe636914e426b411b997208a9ac52dd9d103b2538c2: Status 404 returned error can't find the container with id 76b8281a2324ae2d7f5ccbe636914e426b411b997208a9ac52dd9d103b2538c2 Sep 14 01:51:51.511002 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.510958 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:51:51.511161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.511019 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 14 01:51:51.613357 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.613240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" event={"ID":"0465dd4d-0dfe-4768-96ca-1a76405f5f90","Type":"ContainerStarted","Data":"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28"} Sep 14 01:51:51.613357 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.613317 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" event={"ID":"0465dd4d-0dfe-4768-96ca-1a76405f5f90","Type":"ContainerStarted","Data":"76b8281a2324ae2d7f5ccbe636914e426b411b997208a9ac52dd9d103b2538c2"} Sep 14 01:51:51.631812 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:51.631757 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" podStartSLOduration=1.631742646 podStartE2EDuration="1.631742646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:51.629420332 +0000 UTC m=+1258.302536508" watchObservedRunningTime="2026-09-14 01:51:51.631742646 +0000 UTC m=+1258.304858854" Sep 14 01:51:52.510323 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:52.510285 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:51:52.510735 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:52.510341 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 14 01:51:53.511154 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:53.511114 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Sep 14 01:51:53.511651 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:53.511184 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Sep 14 01:51:54.599226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:54.599185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:54.601012 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:54.600986 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:51:55.141218 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.141183 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:51:55.146289 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.146236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.148486 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.148461 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ff93e\"" Sep 14 01:51:55.148653 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.148633 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7ff93e-dockercfg-kr2gf\"" Sep 14 01:51:55.153208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.153183 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:51:55.177428 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.177379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kz6l\" (UniqueName: \"kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.177572 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.177558 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.278772 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.278734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.278943 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.278826 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kz6l\" (UniqueName: \"kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.279779 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.279750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.286392 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.286361 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kz6l\" (UniqueName: \"kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l\") pod \"payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.459573 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.459478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:55.732384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:55.732359 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:51:55.734144 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:51:55.734103 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eac46a_6714_44ca_bfa7_7876b08f2214.slice/crio-295588f5d7f55126f0b8313bbb71380f13ca4264c969bfc26d927baaff6716f6 WatchSource:0}: Error finding container 295588f5d7f55126f0b8313bbb71380f13ca4264c969bfc26d927baaff6716f6: Status 404 returned error can't find the container with id 295588f5d7f55126f0b8313bbb71380f13ca4264c969bfc26d927baaff6716f6 Sep 14 01:51:56.641860 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:56.641827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" event={"ID":"88eac46a-6714-44ca-bfa7-7876b08f2214","Type":"ContainerStarted","Data":"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969"} Sep 14 01:51:56.642076 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:56.641882 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" event={"ID":"88eac46a-6714-44ca-bfa7-7876b08f2214","Type":"ContainerStarted","Data":"295588f5d7f55126f0b8313bbb71380f13ca4264c969bfc26d927baaff6716f6"} Sep 14 01:51:56.642076 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:56.641956 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:51:56.659051 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:51:56.658989 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" podStartSLOduration=1.658975284 podStartE2EDuration="1.658975284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:51:56.65570649 +0000 UTC m=+1263.328822855" watchObservedRunningTime="2026-09-14 01:51:56.658975284 +0000 UTC m=+1263.332091459" Sep 14 01:52:01.038628 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:01.038592 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:52:01.039088 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:01.038847 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:52:01.051683 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:01.051660 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:52:01.678404 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:01.678371 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:52:07.649390 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:07.649362 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:52:15.748356 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.748273 2573 generic.go:356] "Generic (PLEG): container finished" podID="72585742-6b07-4546-9a88-b4da4c187214" containerID="ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0" exitCode=0 Sep 14 01:52:15.748356 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.748326 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerDied","Data":"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0"} Sep 14 01:52:15.748821 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.748806 2573 scope.go:117] "RemoveContainer" containerID="ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0" Sep 14 01:52:15.749951 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.749926 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:15.750019 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.750005 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:15.750059 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:15.750041 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:16.755678 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:16.755634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerStarted","Data":"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9"} Sep 14 01:52:17.321162 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:17.321123 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:52:17.322217 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:17.322192 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 14 01:52:17.322396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:17.322281 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 14 01:52:18.321760 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:18.321727 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 14 01:52:18.322166 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:18.321784 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 14 01:52:19.321236 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:19.321201 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 14 01:52:19.321438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:19.321303 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 14 01:52:19.595501 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:19.595408 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:52:20.321157 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:20.321116 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 14 01:52:20.321378 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:20.321194 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 14 01:52:21.321276 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:21.321218 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Sep 14 01:52:21.321664 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:21.321342 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Sep 14 01:52:22.325537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:22.325503 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:52:22.326544 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:22.326522 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:52:30.010355 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.007958 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:52:30.014686 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.014656 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.017976 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.017898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-000f34a9-openshift-default-dockercfg-2mn4h\"" Sep 14 01:52:30.021621 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.021593 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:52:30.147551 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnqs\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147724 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147572 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147724 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147662 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147724 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147842 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147948 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147852 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.147948 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.147920 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249024 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.248997 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249186 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249186 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249186 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249186 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249132 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249303 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249419 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnqs\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249647 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249748 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249687 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249833 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249815 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.249985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249967 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.250056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.249978 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.251615 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.251588 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.251833 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.251813 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.256828 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.256796 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.256952 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.256935 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnqs\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs\") pod \"e2e-mt-000f34a9-openshift-default-578664bb88-chl5t\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.329238 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.329128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:30.605688 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.605663 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:52:30.607324 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:52:30.607296 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c71451_d9a3_4905_b7fe_67969b7723c5.slice/crio-2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a WatchSource:0}: Error finding container 2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a: Status 404 returned error can't find the container with id 2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a Sep 14 01:52:30.609921 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.609887 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:30.610006 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.609959 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:30.610006 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.609986 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:52:30.846048 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.846004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerStarted","Data":"b5e00cca69af3066275c9035f0b8f767b29e1f11f8bde10f650cf958f21952f5"} Sep 14 01:52:30.846237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.846068 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerStarted","Data":"2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a"} Sep 14 01:52:30.864076 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:30.863974 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podStartSLOduration=1.8639567860000001 podStartE2EDuration="1.863956786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:52:30.862186446 +0000 UTC m=+1297.535302625" watchObservedRunningTime="2026-09-14 01:52:30.863956786 +0000 UTC m=+1297.537072961" Sep 14 01:52:31.329512 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:31.329468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:31.334946 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:31.334919 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:31.852342 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:31.852312 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:52:31.853357 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:52:31.853337 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:53:08.046901 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.046866 2573 generic.go:356] "Generic (PLEG): container finished" podID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerID="b5e00cca69af3066275c9035f0b8f767b29e1f11f8bde10f650cf958f21952f5" exitCode=0 Sep 14 01:53:08.047451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.046948 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerDied","Data":"b5e00cca69af3066275c9035f0b8f767b29e1f11f8bde10f650cf958f21952f5"} Sep 14 01:53:08.047573 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.047556 2573 scope.go:117] "RemoveContainer" containerID="b5e00cca69af3066275c9035f0b8f767b29e1f11f8bde10f650cf958f21952f5" Sep 14 01:53:08.048876 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.048844 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:08.048978 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.048932 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:08.049023 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:08.048977 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:09.053840 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:09.053807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerStarted","Data":"5830055cb7d0dee513ab270d0209ec00c773d5b0445c4ec7bfadaf8059aaecab"} Sep 14 01:53:09.329636 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:09.329541 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:53:09.329837 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:09.329814 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:09.329897 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:09.329877 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:10.330204 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:10.330165 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:10.330634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:10.330230 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:11.329950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:11.329912 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:11.330132 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:11.329973 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:12.330194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:12.330157 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:12.330627 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:12.330230 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:13.330472 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:13.330436 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:13.330860 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:13.330507 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:14.330379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:14.330336 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 01:53:14.330542 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:14.330403 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 01:53:15.334683 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:15.334652 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:53:15.335082 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:15.334998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:53:15.335658 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:15.335640 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:53:28.240402 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.240367 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:53:28.240880 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.240653 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" podUID="97e83eda-6b57-4d54-a01f-0d717807c725" containerName="maas-api" containerID="cri-o://4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d" gracePeriod=30 Sep 14 01:53:28.311781 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.311747 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:53:28.312049 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.312018 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" podUID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" containerName="payload-pre-processing" containerID="cri-o://d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8" gracePeriod=30 Sep 14 01:53:28.616221 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.616198 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:53:28.619384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.619365 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:53:28.674793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674766 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume\") pod \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " Sep 14 01:53:28.674985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674838 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls\") pod \"97e83eda-6b57-4d54-a01f-0d717807c725\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " Sep 14 01:53:28.674985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674891 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94kwg\" (UniqueName: \"kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg\") pod \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\" (UID: \"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718\") " Sep 14 01:53:28.674985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674915 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfg6r\" (UniqueName: \"kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r\") pod \"97e83eda-6b57-4d54-a01f-0d717807c725\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " Sep 14 01:53:28.674985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674943 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs\") pod \"97e83eda-6b57-4d54-a01f-0d717807c725\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " Sep 14 01:53:28.674985 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.674982 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca\") pod \"97e83eda-6b57-4d54-a01f-0d717807c725\" (UID: \"97e83eda-6b57-4d54-a01f-0d717807c725\") " Sep 14 01:53:28.675237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.675203 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" (UID: "0a5ded47-51f6-4eef-a8c9-2e7e6efbe718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:28.675451 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.675429 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.675778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.675724 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97e83eda-6b57-4d54-a01f-0d717807c725" (UID: "97e83eda-6b57-4d54-a01f-0d717807c725"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:28.677415 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.677387 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg" (OuterVolumeSpecName: "kube-api-access-94kwg") pod "0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" (UID: "0a5ded47-51f6-4eef-a8c9-2e7e6efbe718"). InnerVolumeSpecName "kube-api-access-94kwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:28.677606 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.677584 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97e83eda-6b57-4d54-a01f-0d717807c725" (UID: "97e83eda-6b57-4d54-a01f-0d717807c725"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:53:28.677987 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.677960 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r" (OuterVolumeSpecName: "kube-api-access-gfg6r") pod "97e83eda-6b57-4d54-a01f-0d717807c725" (UID: "97e83eda-6b57-4d54-a01f-0d717807c725"). InnerVolumeSpecName "kube-api-access-gfg6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:28.678094 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.678045 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97e83eda-6b57-4d54-a01f-0d717807c725" (UID: "97e83eda-6b57-4d54-a01f-0d717807c725"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:53:28.777032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.776938 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.777032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.776987 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94kwg\" (UniqueName: \"kubernetes.io/projected/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718-kube-api-access-94kwg\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.777032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.777002 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfg6r\" (UniqueName: \"kubernetes.io/projected/97e83eda-6b57-4d54-a01f-0d717807c725-kube-api-access-gfg6r\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.777032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.777019 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e83eda-6b57-4d54-a01f-0d717807c725-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:28.777032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:28.777034 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e83eda-6b57-4d54-a01f-0d717807c725-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:29.173229 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.173192 2573 generic.go:356] "Generic (PLEG): container finished" podID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" containerID="d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8" exitCode=0 Sep 14 01:53:29.173407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.173241 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" Sep 14 01:53:29.173407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.173292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" event={"ID":"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718","Type":"ContainerDied","Data":"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8"} Sep 14 01:53:29.173407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.173355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb" event={"ID":"0a5ded47-51f6-4eef-a8c9-2e7e6efbe718","Type":"ContainerDied","Data":"94f980af43a977d1556d64b67183ba8ee28bb7f14471ef29118bc337e1192361"} Sep 14 01:53:29.173407 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.173385 2573 scope.go:117] "RemoveContainer" containerID="d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8" Sep 14 01:53:29.174855 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.174825 2573 generic.go:356] "Generic (PLEG): container finished" podID="97e83eda-6b57-4d54-a01f-0d717807c725" containerID="4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d" exitCode=0 Sep 14 01:53:29.174971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.174861 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" event={"ID":"97e83eda-6b57-4d54-a01f-0d717807c725","Type":"ContainerDied","Data":"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d"} Sep 14 01:53:29.174971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.174886 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" Sep 14 01:53:29.174971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.174923 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j" event={"ID":"97e83eda-6b57-4d54-a01f-0d717807c725","Type":"ContainerDied","Data":"f0e744f2016437c2c0aac07bce68c2137ec4388e658ddaf85273fabb4f24cff3"} Sep 14 01:53:29.184167 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.184150 2573 scope.go:117] "RemoveContainer" containerID="d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8" Sep 14 01:53:29.184523 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:53:29.184475 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8\": container with ID starting with d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8 not found: ID does not exist" containerID="d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8" Sep 14 01:53:29.184622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.184533 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8"} err="failed to get container status \"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8\": rpc error: code = NotFound desc = could not find container \"d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8\": container with ID starting with d1a1fa06d60363d931757e6ea5e5c0bb46c64891094d68b488e55859f72e61d8 not found: ID does not exist" Sep 14 01:53:29.184622 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.184578 2573 scope.go:117] "RemoveContainer" containerID="4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d" Sep 14 01:53:29.192967 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.192953 2573 scope.go:117] "RemoveContainer" containerID="4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d" Sep 14 01:53:29.193194 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:53:29.193179 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d\": container with ID starting with 4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d not found: ID does not exist" containerID="4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d" Sep 14 01:53:29.193237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.193199 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d"} err="failed to get container status \"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d\": rpc error: code = NotFound desc = could not find container \"4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d\": container with ID starting with 4e8681d83b61997b917c4fedc318a6c8e722665601840fa190de80db2210955d not found: ID does not exist" Sep 14 01:53:29.199928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.199903 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:53:29.201991 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.201970 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a36bf3-79bb585f97-bss4j"] Sep 14 01:53:29.210474 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.210454 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:53:29.217916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.217887 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a36bf3-796d9fcd74-kzsqb"] Sep 14 01:53:29.862673 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.862643 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" path="/var/lib/kubelet/pods/0a5ded47-51f6-4eef-a8c9-2e7e6efbe718/volumes" Sep 14 01:53:29.863035 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:29.863006 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e83eda-6b57-4d54-a01f-0d717807c725" path="/var/lib/kubelet/pods/97e83eda-6b57-4d54-a01f-0d717807c725/volumes" Sep 14 01:53:34.971093 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:34.971059 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:53:34.971681 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:34.971275 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" containerID="cri-o://2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f" gracePeriod=30 Sep 14 01:53:36.556825 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:36.556784 2573 patch_prober.go:28] interesting pod/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:53:36.557303 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:36.556870 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:53:40.224872 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.224844 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:53:40.236405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.236336 2573 generic.go:356] "Generic (PLEG): container finished" podID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerID="2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f" exitCode=0 Sep 14 01:53:40.236405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.236389 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" Sep 14 01:53:40.236584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.236412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerDied","Data":"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f"} Sep 14 01:53:40.236584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.236465 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq" event={"ID":"77d120a9-ebf2-459b-a251-5341f9ed383b","Type":"ContainerDied","Data":"06aed4017286e79a1fdb1911d60a3c2966860084797d4828d31bcdd38982e3c8"} Sep 14 01:53:40.236584 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.236489 2573 scope.go:117] "RemoveContainer" containerID="2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f" Sep 14 01:53:40.252005 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.250878 2573 scope.go:117] "RemoveContainer" containerID="f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40" Sep 14 01:53:40.262795 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.262772 2573 scope.go:117] "RemoveContainer" containerID="2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f" Sep 14 01:53:40.263141 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:53:40.263116 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f\": container with ID starting with 2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f not found: ID does not exist" containerID="2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f" Sep 14 01:53:40.263199 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.263155 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f"} err="failed to get container status \"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f\": rpc error: code = NotFound desc = could not find container \"2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f\": container with ID starting with 2dcc196869459ee9f8093135f697748f548e91d8e206d89f0f548ae3e7c8440f not found: ID does not exist" Sep 14 01:53:40.263199 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.263181 2573 scope.go:117] "RemoveContainer" containerID="f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40" Sep 14 01:53:40.263491 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:53:40.263475 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40\": container with ID starting with f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40 not found: ID does not exist" containerID="f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40" Sep 14 01:53:40.263562 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.263493 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40"} err="failed to get container status \"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40\": rpc error: code = NotFound desc = could not find container \"f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40\": container with ID starting with f91334c2f79b502aadbe71c6e080aeeacd5a555781e2210596f38347583d5c40 not found: ID does not exist" Sep 14 01:53:40.406395 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406362 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406405 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406423 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pz6w\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406453 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406479 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406563 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406535 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406577 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406621 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406669 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert\") pod \"77d120a9-ebf2-459b-a251-5341f9ed383b\" (UID: \"77d120a9-ebf2-459b-a251-5341f9ed383b\") " Sep 14 01:53:40.406816 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406752 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:53:40.407113 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.406815 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:53:40.407113 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407065 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data" (OuterVolumeSpecName: "istio-data") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:53:40.407113 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407075 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:40.407113 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407125 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:53:40.407503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407408 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.407503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407429 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.407503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407443 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d120a9-ebf2-459b-a251-5341f9ed383b-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.407503 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.407456 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.408908 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.408879 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:53:40.409014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.408925 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token" (OuterVolumeSpecName: "istio-token") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:40.409014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.408953 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w" (OuterVolumeSpecName: "kube-api-access-8pz6w") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "kube-api-access-8pz6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:40.409014 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.408971 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77d120a9-ebf2-459b-a251-5341f9ed383b" (UID: "77d120a9-ebf2-459b-a251-5341f9ed383b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:53:40.508308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.508223 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.508308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.508274 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pz6w\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-kube-api-access-8pz6w\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.508308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.508289 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.508308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.508297 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.508308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.508304 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d120a9-ebf2-459b-a251-5341f9ed383b-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:40.558025 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.557993 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:53:40.562479 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:40.562454 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a36bf3-openshift-default-6fd54959fb-qchtq"] Sep 14 01:53:41.863410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:41.863378 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" path="/var/lib/kubelet/pods/77d120a9-ebf2-459b-a251-5341f9ed383b/volumes" Sep 14 01:53:45.956068 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:45.956038 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:53:45.956537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:45.956306 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" podUID="041e597c-2ea8-4e14-b195-21a039875ce0" containerName="payload-pre-processing" containerID="cri-o://ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0" gracePeriod=30 Sep 14 01:53:46.236364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.236342 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:53:46.272180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.272149 2573 generic.go:356] "Generic (PLEG): container finished" podID="041e597c-2ea8-4e14-b195-21a039875ce0" containerID="ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0" exitCode=0 Sep 14 01:53:46.272372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.272200 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" Sep 14 01:53:46.272372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.272238 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" event={"ID":"041e597c-2ea8-4e14-b195-21a039875ce0","Type":"ContainerDied","Data":"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0"} Sep 14 01:53:46.272372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.272313 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z" event={"ID":"041e597c-2ea8-4e14-b195-21a039875ce0","Type":"ContainerDied","Data":"64d7299ef6264da908156a40d95cb1708d3ea37b268cb16e9f21bc31a208c0a3"} Sep 14 01:53:46.272372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.272342 2573 scope.go:117] "RemoveContainer" containerID="ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0" Sep 14 01:53:46.283149 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.283126 2573 scope.go:117] "RemoveContainer" containerID="ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0" Sep 14 01:53:46.283525 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:53:46.283498 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0\": container with ID starting with ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0 not found: ID does not exist" containerID="ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0" Sep 14 01:53:46.283629 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.283535 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0"} err="failed to get container status \"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0\": rpc error: code = NotFound desc = could not find container \"ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0\": container with ID starting with ffd6581ef5aa1c4211f42ecda620c415507a82b5d0421387b21ce43703c8cff0 not found: ID does not exist" Sep 14 01:53:46.367940 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.367907 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82bl6\" (UniqueName: \"kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6\") pod \"041e597c-2ea8-4e14-b195-21a039875ce0\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " Sep 14 01:53:46.368147 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.368032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume\") pod \"041e597c-2ea8-4e14-b195-21a039875ce0\" (UID: \"041e597c-2ea8-4e14-b195-21a039875ce0\") " Sep 14 01:53:46.368370 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.368340 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "041e597c-2ea8-4e14-b195-21a039875ce0" (UID: "041e597c-2ea8-4e14-b195-21a039875ce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:53:46.370244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.370216 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6" (OuterVolumeSpecName: "kube-api-access-82bl6") pod "041e597c-2ea8-4e14-b195-21a039875ce0" (UID: "041e597c-2ea8-4e14-b195-21a039875ce0"). InnerVolumeSpecName "kube-api-access-82bl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:53:46.469545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.469473 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/041e597c-2ea8-4e14-b195-21a039875ce0-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:46.469545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.469498 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bl6\" (UniqueName: \"kubernetes.io/projected/041e597c-2ea8-4e14-b195-21a039875ce0-kube-api-access-82bl6\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:53:46.594468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.594437 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:53:46.599084 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:46.599058 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-321811-6bd5b9789d-r2r5z"] Sep 14 01:53:47.862439 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:47.862395 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041e597c-2ea8-4e14-b195-21a039875ce0" path="/var/lib/kubelet/pods/041e597c-2ea8-4e14-b195-21a039875ce0/volumes" Sep 14 01:53:48.796113 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796076 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:53:48.796759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796731 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" containerName="payload-pre-processing" Sep 14 01:53:48.796759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796751 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" containerName="payload-pre-processing" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796773 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e83eda-6b57-4d54-a01f-0d717807c725" containerName="maas-api" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796778 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e83eda-6b57-4d54-a01f-0d717807c725" containerName="maas-api" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796787 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796793 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796801 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041e597c-2ea8-4e14-b195-21a039875ce0" containerName="payload-pre-processing" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796807 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="041e597c-2ea8-4e14-b195-21a039875ce0" containerName="payload-pre-processing" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796815 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796820 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796880 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5ded47-51f6-4eef-a8c9-2e7e6efbe718" containerName="payload-pre-processing" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796887 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="041e597c-2ea8-4e14-b195-21a039875ce0" containerName="payload-pre-processing" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796894 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:48.797091 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.796901 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e83eda-6b57-4d54-a01f-0d717807c725" containerName="maas-api" Sep 14 01:53:48.801397 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.801377 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.805147 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.805115 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-000f34a9\"/\"default-dockercfg-h9s8p\"" Sep 14 01:53:48.805475 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.805415 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-000f34a9\"/\"kube-root-ca.crt\"" Sep 14 01:53:48.806214 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.806188 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-000f34a9\"/\"e2e-lifecycle-model-000f34a9-kserve-self-signed-certs\"" Sep 14 01:53:48.806461 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.806442 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-000f34a9\"/\"openshift-service-ca.crt\"" Sep 14 01:53:48.809963 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.809938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:53:48.895487 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.895487 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.895927 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.895927 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895575 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkph\" (UniqueName: \"kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.895927 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.895927 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.895709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997080 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997296 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkph\" (UniqueName: \"kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997487 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997463 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997557 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997502 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.997618 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.997575 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.999600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.999579 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:48.999826 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:48.999809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:49.004833 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:49.004809 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkph\" (UniqueName: \"kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph\") pod \"e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:49.115701 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:49.115618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:49.391515 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:49.391477 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:53:49.393519 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:53:49.393484 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cdbc18c_787d_403c_ac04_42fbebc004cc.slice/crio-88d67606362bd66732f5de629753fad6772fd635956cd6b97d8892710f640cfb WatchSource:0}: Error finding container 88d67606362bd66732f5de629753fad6772fd635956cd6b97d8892710f640cfb: Status 404 returned error can't find the container with id 88d67606362bd66732f5de629753fad6772fd635956cd6b97d8892710f640cfb Sep 14 01:53:50.301669 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:50.301632 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" event={"ID":"2cdbc18c-787d-403c-ac04-42fbebc004cc","Type":"ContainerStarted","Data":"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28"} Sep 14 01:53:50.302102 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:50.301678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" event={"ID":"2cdbc18c-787d-403c-ac04-42fbebc004cc","Type":"ContainerStarted","Data":"88d67606362bd66732f5de629753fad6772fd635956cd6b97d8892710f640cfb"} Sep 14 01:53:50.318512 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:50.318458 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" podStartSLOduration=2.318442902 podStartE2EDuration="2.318442902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:53:50.316497026 +0000 UTC m=+1376.989613200" watchObservedRunningTime="2026-09-14 01:53:50.318442902 +0000 UTC m=+1376.991559077" Sep 14 01:53:58.356167 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.356131 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 01:53:58.356912 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.356885 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d120a9-ebf2-459b-a251-5341f9ed383b" containerName="istio-proxy" Sep 14 01:53:58.361647 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.361624 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.364081 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.364056 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-5d3ac7-openshift-default-dockercfg-8655z\"" Sep 14 01:53:58.371705 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.371675 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 01:53:58.495735 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495701 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.495928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tb2w\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.495928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.495928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495879 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.495928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495924 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.496062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.495981 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.496062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.496032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.496062 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.496058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.496161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.496085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597137 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tb2w\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597350 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597659 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597659 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597659 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597659 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597559 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597828 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597664 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597861 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.597922 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.597901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.598108 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.598090 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.599841 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.599820 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.599943 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.599869 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.604966 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.604943 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.605072 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.605021 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tb2w\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w\") pod \"e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.675098 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.675069 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:53:58.952049 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.952021 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 01:53:58.953769 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:53:58.953742 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178ecc4b_0e2c_462e_bc77_f86728427097.slice/crio-a539ed54f41cabfebe2589e750949cd810d122963200330ad19662000b6ca1f1 WatchSource:0}: Error finding container a539ed54f41cabfebe2589e750949cd810d122963200330ad19662000b6ca1f1: Status 404 returned error can't find the container with id a539ed54f41cabfebe2589e750949cd810d122963200330ad19662000b6ca1f1 Sep 14 01:53:58.957006 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.956955 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:58.957107 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.957059 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:58.957180 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:58.957104 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:53:59.116276 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.116185 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:59.121273 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.119698 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:59.136237 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.136194 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:59.350733 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.350634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerStarted","Data":"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac"} Sep 14 01:53:59.350733 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.350709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerStarted","Data":"a539ed54f41cabfebe2589e750949cd810d122963200330ad19662000b6ca1f1"} Sep 14 01:53:59.367406 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.367376 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:53:59.369492 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.369447 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podStartSLOduration=1.369432865 podStartE2EDuration="1.369432865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:53:59.368191926 +0000 UTC m=+1386.041308114" watchObservedRunningTime="2026-09-14 01:53:59.369432865 +0000 UTC m=+1386.042549040" Sep 14 01:53:59.675759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:53:59.675727 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:00.681490 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:00.681464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:01.362601 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:01.362569 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:01.363469 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:01.363445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:30.525368 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.525336 2573 generic.go:356] "Generic (PLEG): container finished" podID="178ecc4b-0e2c-462e-bc77-f86728427097" containerID="0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac" exitCode=0 Sep 14 01:54:30.525860 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.525382 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerDied","Data":"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac"} Sep 14 01:54:30.525950 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.525931 2573 scope.go:117] "RemoveContainer" containerID="0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac" Sep 14 01:54:30.527144 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.527117 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:54:30.527308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.527197 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:54:30.527308 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:30.527246 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 01:54:31.363379 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:31.363336 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:31.532945 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:31.532910 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerStarted","Data":"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61"} Sep 14 01:54:31.676154 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:31.676109 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:31.677468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:31.677440 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:31.677751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:31.677721 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:32.676313 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:32.676279 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:32.676890 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:32.676844 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:33.676287 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:33.675843 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:33.676287 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:33.675905 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:34.675778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:34.675740 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:34.675991 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:34.675810 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:35.675715 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:35.675668 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:35.676207 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:35.675753 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:36.676130 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:36.676083 2573 patch_prober.go:28] interesting pod/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 01:54:36.676610 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:36.676175 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 01:54:37.679934 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:37.679902 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:37.680323 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:37.680174 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:37.680925 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:37.680905 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 01:54:39.817886 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.817860 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 01:54:39.821736 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.821703 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:39.823986 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.823961 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-86995c-serving-cert\"" Sep 14 01:54:39.829155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.829123 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 01:54:39.940090 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.940046 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9mg7\" (UniqueName: \"kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:39.940316 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.940101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:39.940316 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.940123 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:39.940454 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:39.940360 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.041974 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.041939 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9mg7\" (UniqueName: \"kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.041974 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.041978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.042212 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.042007 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.042212 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.042048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.042783 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.042759 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.044570 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.044543 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.044656 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.044632 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.049483 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.049461 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9mg7\" (UniqueName: \"kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7\") pod \"maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.134117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.134093 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:40.407354 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.407329 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 01:54:40.410066 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:54:40.410025 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice/crio-8f0b00727187d43ed360af460693ba008c2c92532a44d7cfa6fffbde2e882eec WatchSource:0}: Error finding container 8f0b00727187d43ed360af460693ba008c2c92532a44d7cfa6fffbde2e882eec: Status 404 returned error can't find the container with id 8f0b00727187d43ed360af460693ba008c2c92532a44d7cfa6fffbde2e882eec Sep 14 01:54:40.411699 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.411679 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 01:54:40.605691 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:40.601313 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" event={"ID":"1f212633-1f9c-46d2-be62-ac16807ac837","Type":"ContainerStarted","Data":"8f0b00727187d43ed360af460693ba008c2c92532a44d7cfa6fffbde2e882eec"} Sep 14 01:54:41.608298 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:41.608240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" event={"ID":"1f212633-1f9c-46d2-be62-ac16807ac837","Type":"ContainerStarted","Data":"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96"} Sep 14 01:54:41.608732 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:41.608466 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:54:41.626545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:41.626495 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" podStartSLOduration=2.315622221 podStartE2EDuration="2.626481461s" totalImagesPullingTime="310.85924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:54:41.624343752 +0000 UTC m=+1428.297459929" watchObservedRunningTime="2026-09-14 01:54:41.626481461 +0000 UTC m=+1428.299597635" Sep 14 01:54:47.618900 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:54:47.618830 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 01:55:39.411266 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:39.411215 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:55:54.431232 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.431162 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" podUID="2cdbc18c-787d-403c-ac04-42fbebc004cc" containerName="main" containerID="cri-o://516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28" gracePeriod=45 Sep 14 01:55:54.688107 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.688043 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:55:54.799616 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799581 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.799793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799627 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.799793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799683 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfkph\" (UniqueName: \"kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.799793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799703 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.799793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799752 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.799793 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799780 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir\") pod \"2cdbc18c-787d-403c-ac04-42fbebc004cc\" (UID: \"2cdbc18c-787d-403c-ac04-42fbebc004cc\") " Sep 14 01:55:54.800067 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.799974 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home" (OuterVolumeSpecName: "home") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:55:54.800067 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.800039 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:55:54.800168 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.800102 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:55:54.800518 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.800498 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-model-cache\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.800581 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.800525 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-tmp-dir\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.800581 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.800538 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-home\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.802042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.802011 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm" (OuterVolumeSpecName: "dshm") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:55:54.802042 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.802022 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:55:54.802181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.802068 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph" (OuterVolumeSpecName: "kube-api-access-jfkph") pod "2cdbc18c-787d-403c-ac04-42fbebc004cc" (UID: "2cdbc18c-787d-403c-ac04-42fbebc004cc"). InnerVolumeSpecName "kube-api-access-jfkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:55:54.902066 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.902019 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cdbc18c-787d-403c-ac04-42fbebc004cc-dshm\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.902066 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.902061 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cdbc18c-787d-403c-ac04-42fbebc004cc-tls-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.902066 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.902072 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfkph\" (UniqueName: \"kubernetes.io/projected/2cdbc18c-787d-403c-ac04-42fbebc004cc-kube-api-access-jfkph\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:55:54.997312 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.997198 2573 generic.go:356] "Generic (PLEG): container finished" podID="2cdbc18c-787d-403c-ac04-42fbebc004cc" containerID="516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28" exitCode=0 Sep 14 01:55:54.997312 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.997239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" Sep 14 01:55:54.997312 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.997281 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" event={"ID":"2cdbc18c-787d-403c-ac04-42fbebc004cc","Type":"ContainerDied","Data":"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28"} Sep 14 01:55:54.997565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.997320 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv" event={"ID":"2cdbc18c-787d-403c-ac04-42fbebc004cc","Type":"ContainerDied","Data":"88d67606362bd66732f5de629753fad6772fd635956cd6b97d8892710f640cfb"} Sep 14 01:55:54.997565 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:54.997336 2573 scope.go:117] "RemoveContainer" containerID="516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28" Sep 14 01:55:55.007116 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:55.007101 2573 scope.go:117] "RemoveContainer" containerID="516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28" Sep 14 01:55:55.007488 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:55:55.007470 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28\": container with ID starting with 516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28 not found: ID does not exist" containerID="516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28" Sep 14 01:55:55.007534 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:55.007497 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28"} err="failed to get container status \"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28\": rpc error: code = NotFound desc = could not find container \"516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28\": container with ID starting with 516b823db2da568bf4d88e7fefd8eea9e630df830d416f36c6135259ad57de28 not found: ID does not exist" Sep 14 01:55:55.018314 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:55.018284 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:55:55.021141 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:55.021122 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-000f34a9/e2e-lifecycle-model-000f34a9-kserve-54f569c995-s89zv"] Sep 14 01:55:55.862893 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:55:55.862856 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cdbc18c-787d-403c-ac04-42fbebc004cc" path="/var/lib/kubelet/pods/2cdbc18c-787d-403c-ac04-42fbebc004cc/volumes" Sep 14 01:56:27.967690 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:27.967656 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:56:27.968210 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:27.967922 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" containerID="cri-o://5830055cb7d0dee513ab270d0209ec00c773d5b0445c4ec7bfadaf8059aaecab" gracePeriod=30 Sep 14 01:56:30.337723 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:30.337681 2573 patch_prober.go:28] interesting pod/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:56:30.338101 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:30.337744 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:56:33.209984 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.209950 2573 generic.go:356] "Generic (PLEG): container finished" podID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerID="5830055cb7d0dee513ab270d0209ec00c773d5b0445c4ec7bfadaf8059aaecab" exitCode=0 Sep 14 01:56:33.210352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.210023 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerDied","Data":"5830055cb7d0dee513ab270d0209ec00c773d5b0445c4ec7bfadaf8059aaecab"} Sep 14 01:56:33.210352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.210133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" event={"ID":"47c71451-d9a3-4905-b7fe-67969b7723c5","Type":"ContainerDied","Data":"2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a"} Sep 14 01:56:33.210352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.210161 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2924d1bfb014c632b94a9be81da6b3e17be2def6cfadb25d950b84bd85f49a6a" Sep 14 01:56:33.210352 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.210179 2573 scope.go:117] "RemoveContainer" containerID="b5e00cca69af3066275c9035f0b8f767b29e1f11f8bde10f650cf958f21952f5" Sep 14 01:56:33.226615 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.226594 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:56:33.299757 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299671 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.299757 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.299757 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299758 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299855 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299875 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299889 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299916 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxnqs\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299944 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket\") pod \"47c71451-d9a3-4905-b7fe-67969b7723c5\" (UID: \"47c71451-d9a3-4905-b7fe-67969b7723c5\") " Sep 14 01:56:33.300033 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.299997 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:33.300442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300031 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:56:33.300442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300121 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:33.300442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:33.300442 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300429 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47c71451-d9a3-4905-b7fe-67969b7723c5-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.300635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300452 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.300635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300466 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.300635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300480 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.300635 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.300279 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:33.302164 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.302134 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:56:33.302328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.302207 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:56:33.302328 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.302306 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:56:33.302427 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.302407 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs" (OuterVolumeSpecName: "kube-api-access-hxnqs") pod "47c71451-d9a3-4905-b7fe-67969b7723c5" (UID: "47c71451-d9a3-4905-b7fe-67969b7723c5"). InnerVolumeSpecName "kube-api-access-hxnqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:56:33.401740 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.401708 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.401740 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.401730 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.401740 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.401739 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.401740 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.401747 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47c71451-d9a3-4905-b7fe-67969b7723c5-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:33.401967 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:33.401755 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnqs\" (UniqueName: \"kubernetes.io/projected/47c71451-d9a3-4905-b7fe-67969b7723c5-kube-api-access-hxnqs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:56:34.216874 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:34.216846 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t" Sep 14 01:56:34.247707 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:34.247677 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:56:34.250046 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:34.250022 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-000f34a9-openshift-default-578664bb88-chl5t"] Sep 14 01:56:35.862327 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:56:35.862279 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" path="/var/lib/kubelet/pods/47c71451-d9a3-4905-b7fe-67969b7723c5/volumes" Sep 14 01:57:11.310481 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.310446 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:57:11.311122 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311101 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:11.311171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311128 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:11.311171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311147 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cdbc18c-787d-403c-ac04-42fbebc004cc" containerName="main" Sep 14 01:57:11.311171 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311155 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cdbc18c-787d-403c-ac04-42fbebc004cc" containerName="main" Sep 14 01:57:11.311306 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311293 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:11.311347 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.311312 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cdbc18c-787d-403c-ac04-42fbebc004cc" containerName="main" Sep 14 01:57:11.316561 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.316539 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.319519 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.319494 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-99c616\"/\"embedding-w0-99c616-kserve-self-signed-certs\"" Sep 14 01:57:11.322295 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.322267 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:57:11.374046 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8jg\" (UniqueName: \"kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.374208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374054 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.374208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.374208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.374208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374108 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.374452 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.374217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475197 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475146 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8jg\" (UniqueName: \"kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475209 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475468 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.475982 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.475951 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.476089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.476011 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.476200 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.476171 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.479312 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.479284 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.480482 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.480445 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.483240 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.483217 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8jg\" (UniqueName: \"kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg\") pod \"embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.630405 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.630320 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:11.904359 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:11.904331 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:57:11.906274 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:57:11.906224 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fe5518e_1519_483e_861b_0599a0aff40d.slice/crio-268eb63987ade1e4b982fc9ce0ac6b78c1bd564a9f7e9b01f1fc3332d1a8adb6 WatchSource:0}: Error finding container 268eb63987ade1e4b982fc9ce0ac6b78c1bd564a9f7e9b01f1fc3332d1a8adb6: Status 404 returned error can't find the container with id 268eb63987ade1e4b982fc9ce0ac6b78c1bd564a9f7e9b01f1fc3332d1a8adb6 Sep 14 01:57:12.416631 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:12.416600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" event={"ID":"4fe5518e-1519-483e-861b-0599a0aff40d","Type":"ContainerStarted","Data":"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a"} Sep 14 01:57:12.416631 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:12.416639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" event={"ID":"4fe5518e-1519-483e-861b-0599a0aff40d","Type":"ContainerStarted","Data":"268eb63987ade1e4b982fc9ce0ac6b78c1bd564a9f7e9b01f1fc3332d1a8adb6"} Sep 14 01:57:12.432601 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:12.432558 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" podStartSLOduration=1.432542438 podStartE2EDuration="1.432542438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:12.431084114 +0000 UTC m=+1579.104200289" watchObservedRunningTime="2026-09-14 01:57:12.432542438 +0000 UTC m=+1579.105658613" Sep 14 01:57:21.630624 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:21.630580 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:21.631077 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:21.630860 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:21.643537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:21.643514 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:22.483571 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:22.483545 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:57:28.546797 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.546754 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:57:28.547297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.547234 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:28.547297 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.547281 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:28.547438 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.547412 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c71451-d9a3-4905-b7fe-67969b7723c5" containerName="istio-proxy" Sep 14 01:57:28.550977 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.550952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.553600 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.553580 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b32209-serving-cert\"" Sep 14 01:57:28.562799 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.562550 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:57:28.646494 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.646460 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.646680 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.646513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.646680 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.646530 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.646776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.646683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h25\" (UniqueName: \"kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.748071 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.748037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h25\" (UniqueName: \"kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.748291 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.748094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.748291 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.748127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.748291 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.748143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.748430 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:57:28.748314 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-serving-cert: secret "maas-api-e2e-api-a-b32209-serving-cert" not found Sep 14 01:57:28.748430 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:57:28.748388 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls podName:8cf1cb4e-947c-48de-928f-341c64fa3ddf nodeName:}" failed. No retries permitted until 2026-09-14 01:57:29.248370788 +0000 UTC m=+1595.921486941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls") pod "maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" (UID: "8cf1cb4e-947c-48de-928f-341c64fa3ddf") : secret "maas-api-e2e-api-a-b32209-serving-cert" not found Sep 14 01:57:28.748809 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.748790 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.750822 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.750795 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:28.756466 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:28.756418 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h25\" (UniqueName: \"kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:29.098872 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.098836 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:57:29.102685 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.102669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.105039 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.105021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b32209-dockercfg-lxtdf\"" Sep 14 01:57:29.105126 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.105060 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b32209\"" Sep 14 01:57:29.112168 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.112145 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:57:29.152140 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.152108 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:57:29.155829 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.155814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.167061 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.167039 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:57:29.254776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.254743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5h8\" (UniqueName: \"kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.254957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.254840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:29.254957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.254881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gcl\" (UniqueName: \"kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.254957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.254910 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.254957 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.254945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.257624 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.257593 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") pod \"maas-api-e2e-api-a-b32209-78b68b864c-dt9qw\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:29.356654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.356551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gcl\" (UniqueName: \"kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.356654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.356622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.356892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.356681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.356892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.356781 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5h8\" (UniqueName: \"kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.357577 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.357541 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.357675 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.357577 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.364245 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.364225 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gcl\" (UniqueName: \"kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl\") pod \"payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.364485 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.364464 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5h8\" (UniqueName: \"kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8\") pod \"payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.416881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.416852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:29.466482 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.466449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:29.467604 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.467490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:29.691188 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.691165 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:57:29.692756 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:57:29.692731 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449c7d04_8a8a_4706_b873_f7eaf13e8dbd.slice/crio-f2a2e176ad166cb4d47207c8aa687d0a6ec9f0a1ce13111270b6dedc8127b671 WatchSource:0}: Error finding container f2a2e176ad166cb4d47207c8aa687d0a6ec9f0a1ce13111270b6dedc8127b671: Status 404 returned error can't find the container with id f2a2e176ad166cb4d47207c8aa687d0a6ec9f0a1ce13111270b6dedc8127b671 Sep 14 01:57:29.758528 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.757487 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:57:29.764193 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:29.763278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:57:29.768299 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:57:29.768084 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9e9490_d578_4bda_8f9e_8bba00976835.slice/crio-4537fe952221e8700e3ef49b0d3d5a51a2427e62a78227390d035e157a0bf01d WatchSource:0}: Error finding container 4537fe952221e8700e3ef49b0d3d5a51a2427e62a78227390d035e157a0bf01d: Status 404 returned error can't find the container with id 4537fe952221e8700e3ef49b0d3d5a51a2427e62a78227390d035e157a0bf01d Sep 14 01:57:30.523343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.523298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" event={"ID":"8f9e9490-d578-4bda-8f9e-8bba00976835","Type":"ContainerStarted","Data":"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5"} Sep 14 01:57:30.523739 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.523368 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" event={"ID":"8f9e9490-d578-4bda-8f9e-8bba00976835","Type":"ContainerStarted","Data":"4537fe952221e8700e3ef49b0d3d5a51a2427e62a78227390d035e157a0bf01d"} Sep 14 01:57:30.523739 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.523531 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:57:30.524965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.524942 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" event={"ID":"449c7d04-8a8a-4706-b873-f7eaf13e8dbd","Type":"ContainerStarted","Data":"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95"} Sep 14 01:57:30.525111 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.524985 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:30.525111 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.525010 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" event={"ID":"449c7d04-8a8a-4706-b873-f7eaf13e8dbd","Type":"ContainerStarted","Data":"f2a2e176ad166cb4d47207c8aa687d0a6ec9f0a1ce13111270b6dedc8127b671"} Sep 14 01:57:30.526633 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.526597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" event={"ID":"8cf1cb4e-947c-48de-928f-341c64fa3ddf","Type":"ContainerStarted","Data":"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581"} Sep 14 01:57:30.526763 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.526651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" event={"ID":"8cf1cb4e-947c-48de-928f-341c64fa3ddf","Type":"ContainerStarted","Data":"71e2b6dd338c0aebd1accc01dae01329d411a7290a9113f0b106af5cffe84d5a"} Sep 14 01:57:30.526763 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.526682 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:30.539005 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.538967 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" podStartSLOduration=1.538954065 podStartE2EDuration="1.538954065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:30.538458621 +0000 UTC m=+1597.211574797" watchObservedRunningTime="2026-09-14 01:57:30.538954065 +0000 UTC m=+1597.212070230" Sep 14 01:57:30.553100 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.553057 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" podStartSLOduration=1.553039233 podStartE2EDuration="1.553039233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:30.552096925 +0000 UTC m=+1597.225213102" watchObservedRunningTime="2026-09-14 01:57:30.553039233 +0000 UTC m=+1597.226155409" Sep 14 01:57:30.567971 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:30.567917 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" podStartSLOduration=2.203991934 podStartE2EDuration="2.567897766s" totalImagesPullingTime="363.905832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:57:30.566693118 +0000 UTC m=+1597.239809298" watchObservedRunningTime="2026-09-14 01:57:30.567897766 +0000 UTC m=+1597.241013942" Sep 14 01:57:36.538507 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:36.538480 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:57:41.534356 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:41.534322 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:57:41.534834 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:57:41.534400 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:58:04.127632 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.127597 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:58:04.132434 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.132417 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.135206 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.135186 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-318a3c-dockercfg-jwgcz\"" Sep 14 01:58:04.135340 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.135207 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-318a3c\"" Sep 14 01:58:04.143173 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.143151 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:58:04.209435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.209401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.209435 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.209438 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqhn\" (UniqueName: \"kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.310245 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.310199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.310245 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.310280 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqhn\" (UniqueName: \"kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.310975 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.310955 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.317769 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.317743 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqhn\" (UniqueName: \"kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn\") pod \"payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.444621 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.444587 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:04.725543 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:04.725514 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:58:04.726963 ip-10-0-132-205 kubenswrapper[2573]: W0914 01:58:04.726939 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40995596_4214_4c7c_a88c_d8a1a7cd2ac1.slice/crio-69ec3b2c271bbc46256a65fc4b0635d97dda7f9f3e6ddf42584d0b8407e4f47a WatchSource:0}: Error finding container 69ec3b2c271bbc46256a65fc4b0635d97dda7f9f3e6ddf42584d0b8407e4f47a: Status 404 returned error can't find the container with id 69ec3b2c271bbc46256a65fc4b0635d97dda7f9f3e6ddf42584d0b8407e4f47a Sep 14 01:58:05.718436 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:05.718394 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" event={"ID":"40995596-4214-4c7c-a88c-d8a1a7cd2ac1","Type":"ContainerStarted","Data":"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1"} Sep 14 01:58:05.718916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:05.718548 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:05.718916 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:05.718596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" event={"ID":"40995596-4214-4c7c-a88c-d8a1a7cd2ac1","Type":"ContainerStarted","Data":"69ec3b2c271bbc46256a65fc4b0635d97dda7f9f3e6ddf42584d0b8407e4f47a"} Sep 14 01:58:05.733181 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:05.733127 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" podStartSLOduration=1.733112351 podStartE2EDuration="1.733112351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 01:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 01:58:05.732706686 +0000 UTC m=+1632.405822862" watchObservedRunningTime="2026-09-14 01:58:05.733112351 +0000 UTC m=+1632.406228526" Sep 14 01:58:16.725372 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:16.725341 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:58:47.121290 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.121227 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:58:47.121864 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.121505 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" podUID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" containerName="maas-api" containerID="cri-o://7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581" gracePeriod=30 Sep 14 01:58:47.177384 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.177352 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:58:47.177663 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.177634 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" podUID="8f9e9490-d578-4bda-8f9e-8bba00976835" containerName="payload-processing" containerID="cri-o://7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5" gracePeriod=30 Sep 14 01:58:47.187165 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.187118 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:58:47.187495 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.187436 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" podUID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" containerName="payload-pre-processing" containerID="cri-o://ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95" gracePeriod=30 Sep 14 01:58:47.518032 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.518009 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:58:47.537482 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.537379 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:58:47.542538 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.542459 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:58:47.669866 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.669774 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume\") pod \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " Sep 14 01:58:47.669866 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.669839 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28h25\" (UniqueName: \"kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25\") pod \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " Sep 14 01:58:47.670089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.669902 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume\") pod \"8f9e9490-d578-4bda-8f9e-8bba00976835\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " Sep 14 01:58:47.670089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.669935 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6gcl\" (UniqueName: \"kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl\") pod \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\" (UID: \"449c7d04-8a8a-4706-b873-f7eaf13e8dbd\") " Sep 14 01:58:47.670089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.669998 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts5h8\" (UniqueName: \"kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8\") pod \"8f9e9490-d578-4bda-8f9e-8bba00976835\" (UID: \"8f9e9490-d578-4bda-8f9e-8bba00976835\") " Sep 14 01:58:47.670089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670031 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") pod \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " Sep 14 01:58:47.670089 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670071 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs\") pod \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " Sep 14 01:58:47.670399 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670177 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca\") pod \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\" (UID: \"8cf1cb4e-947c-48de-928f-341c64fa3ddf\") " Sep 14 01:58:47.670399 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670226 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f9e9490-d578-4bda-8f9e-8bba00976835" (UID: "8f9e9490-d578-4bda-8f9e-8bba00976835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:47.670505 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670461 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "449c7d04-8a8a-4706-b873-f7eaf13e8dbd" (UID: "449c7d04-8a8a-4706-b873-f7eaf13e8dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:47.670757 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670733 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8cf1cb4e-947c-48de-928f-341c64fa3ddf" (UID: "8cf1cb4e-947c-48de-928f-341c64fa3ddf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:58:47.670818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670758 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.670818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.670777 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9e9490-d578-4bda-8f9e-8bba00976835-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.672380 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.672334 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl" (OuterVolumeSpecName: "kube-api-access-m6gcl") pod "449c7d04-8a8a-4706-b873-f7eaf13e8dbd" (UID: "449c7d04-8a8a-4706-b873-f7eaf13e8dbd"). InnerVolumeSpecName "kube-api-access-m6gcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:58:47.672728 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.672707 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25" (OuterVolumeSpecName: "kube-api-access-28h25") pod "8cf1cb4e-947c-48de-928f-341c64fa3ddf" (UID: "8cf1cb4e-947c-48de-928f-341c64fa3ddf"). InnerVolumeSpecName "kube-api-access-28h25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:58:47.672928 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.672911 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8cf1cb4e-947c-48de-928f-341c64fa3ddf" (UID: "8cf1cb4e-947c-48de-928f-341c64fa3ddf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:58:47.673023 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.672999 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8cf1cb4e-947c-48de-928f-341c64fa3ddf" (UID: "8cf1cb4e-947c-48de-928f-341c64fa3ddf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:58:47.673155 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.673137 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8" (OuterVolumeSpecName: "kube-api-access-ts5h8") pod "8f9e9490-d578-4bda-8f9e-8bba00976835" (UID: "8f9e9490-d578-4bda-8f9e-8bba00976835"). InnerVolumeSpecName "kube-api-access-ts5h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:58:47.771543 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771511 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gcl\" (UniqueName: \"kubernetes.io/projected/449c7d04-8a8a-4706-b873-f7eaf13e8dbd-kube-api-access-m6gcl\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.771543 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771538 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts5h8\" (UniqueName: \"kubernetes.io/projected/8f9e9490-d578-4bda-8f9e-8bba00976835-kube-api-access-ts5h8\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.771543 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771548 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.771808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771559 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf1cb4e-947c-48de-928f-341c64fa3ddf-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.771808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771567 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf1cb4e-947c-48de-928f-341c64fa3ddf-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.771808 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.771575 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h25\" (UniqueName: \"kubernetes.io/projected/8cf1cb4e-947c-48de-928f-341c64fa3ddf-kube-api-access-28h25\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:58:47.961194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.961097 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f9e9490-d578-4bda-8f9e-8bba00976835" containerID="7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5" exitCode=0 Sep 14 01:58:47.961194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.961132 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" event={"ID":"8f9e9490-d578-4bda-8f9e-8bba00976835","Type":"ContainerDied","Data":"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5"} Sep 14 01:58:47.961194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.961149 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" Sep 14 01:58:47.961194 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.961189 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd" event={"ID":"8f9e9490-d578-4bda-8f9e-8bba00976835","Type":"ContainerDied","Data":"4537fe952221e8700e3ef49b0d3d5a51a2427e62a78227390d035e157a0bf01d"} Sep 14 01:58:47.961537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.961221 2573 scope.go:117] "RemoveContainer" containerID="7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5" Sep 14 01:58:47.962634 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.962609 2573 generic.go:356] "Generic (PLEG): container finished" podID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" containerID="ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95" exitCode=0 Sep 14 01:58:47.962751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.962653 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" Sep 14 01:58:47.962751 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.962710 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" event={"ID":"449c7d04-8a8a-4706-b873-f7eaf13e8dbd","Type":"ContainerDied","Data":"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95"} Sep 14 01:58:47.962873 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.962758 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t" event={"ID":"449c7d04-8a8a-4706-b873-f7eaf13e8dbd","Type":"ContainerDied","Data":"f2a2e176ad166cb4d47207c8aa687d0a6ec9f0a1ce13111270b6dedc8127b671"} Sep 14 01:58:47.964545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.964519 2573 generic.go:356] "Generic (PLEG): container finished" podID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" containerID="7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581" exitCode=0 Sep 14 01:58:47.964657 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.964557 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" Sep 14 01:58:47.964657 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.964602 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" event={"ID":"8cf1cb4e-947c-48de-928f-341c64fa3ddf","Type":"ContainerDied","Data":"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581"} Sep 14 01:58:47.964657 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.964651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw" event={"ID":"8cf1cb4e-947c-48de-928f-341c64fa3ddf","Type":"ContainerDied","Data":"71e2b6dd338c0aebd1accc01dae01329d411a7290a9113f0b106af5cffe84d5a"} Sep 14 01:58:47.972202 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.972185 2573 scope.go:117] "RemoveContainer" containerID="7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5" Sep 14 01:58:47.972527 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:58:47.972505 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5\": container with ID starting with 7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5 not found: ID does not exist" containerID="7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5" Sep 14 01:58:47.972624 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.972532 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5"} err="failed to get container status \"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5\": rpc error: code = NotFound desc = could not find container \"7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5\": container with ID starting with 7ad62b5fe7bd3c3920be4c333e407785e64c8ac745659c4d180c59958a9006d5 not found: ID does not exist" Sep 14 01:58:47.972624 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.972547 2573 scope.go:117] "RemoveContainer" containerID="ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95" Sep 14 01:58:47.980728 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.980685 2573 scope.go:117] "RemoveContainer" containerID="ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95" Sep 14 01:58:47.980971 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:58:47.980946 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95\": container with ID starting with ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95 not found: ID does not exist" containerID="ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95" Sep 14 01:58:47.981056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.980978 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95"} err="failed to get container status \"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95\": rpc error: code = NotFound desc = could not find container \"ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95\": container with ID starting with ee38eed9d095c7e4620fa4c6b7310f83b68713cb3d16babb84b3f6d28314bb95 not found: ID does not exist" Sep 14 01:58:47.981056 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.980993 2573 scope.go:117] "RemoveContainer" containerID="7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581" Sep 14 01:58:47.983767 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.983748 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:58:47.987177 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.987153 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b32209-78b68b864c-dt9qw"] Sep 14 01:58:47.989881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.989866 2573 scope.go:117] "RemoveContainer" containerID="7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581" Sep 14 01:58:47.990139 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:58:47.990120 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581\": container with ID starting with 7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581 not found: ID does not exist" containerID="7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581" Sep 14 01:58:47.990177 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.990147 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581"} err="failed to get container status \"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581\": rpc error: code = NotFound desc = could not find container \"7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581\": container with ID starting with 7ecdadee3d638c38d2030c923634a1b5865c5b5f5f0cc4c30e2e7c2e062e6581 not found: ID does not exist" Sep 14 01:58:47.996995 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:47.996975 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:58:48.000026 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:48.000004 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b32209-ff6c8d958-69l8t"] Sep 14 01:58:48.010146 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:48.010125 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:58:48.015500 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:48.015481 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b32209-7cd697c659-p2sqd"] Sep 14 01:58:49.863542 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:49.863503 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" path="/var/lib/kubelet/pods/449c7d04-8a8a-4706-b873-f7eaf13e8dbd/volumes" Sep 14 01:58:49.863988 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:49.863972 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" path="/var/lib/kubelet/pods/8cf1cb4e-947c-48de-928f-341c64fa3ddf/volumes" Sep 14 01:58:49.864343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:49.864328 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9e9490-d578-4bda-8f9e-8bba00976835" path="/var/lib/kubelet/pods/8f9e9490-d578-4bda-8f9e-8bba00976835/volumes" Sep 14 01:58:54.551833 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:54.551800 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:58:54.561764 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:58:54.561732 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:59:06.000945 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.000906 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:59:06.001761 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.001721 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" podUID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" containerName="maas-api" containerID="cri-o://0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c" gracePeriod=30 Sep 14 01:59:06.266496 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.266427 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:59:06.361327 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.361291 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") pod \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " Sep 14 01:59:06.361498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.361356 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs\") pod \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " Sep 14 01:59:06.361498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.361381 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpc8v\" (UniqueName: \"kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v\") pod \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " Sep 14 01:59:06.361498 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.361417 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca\") pod \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\" (UID: \"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf\") " Sep 14 01:59:06.361829 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.361797 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" (UID: "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:06.363657 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.363634 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" (UID: "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:59:06.364121 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.364104 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v" (OuterVolumeSpecName: "kube-api-access-cpc8v") pod "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" (UID: "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf"). InnerVolumeSpecName "kube-api-access-cpc8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:06.364121 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.364106 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" (UID: "7c44ce5d-4f0a-40d1-8f98-dadee2624fbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:59:06.462759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.462723 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.462759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.462753 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.462759 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.462762 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpc8v\" (UniqueName: \"kubernetes.io/projected/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-kube-api-access-cpc8v\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:06.462996 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:06.462773 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:07.075130 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.075094 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" containerID="0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c" exitCode=0 Sep 14 01:59:07.075619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.075166 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" Sep 14 01:59:07.075619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.075166 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" event={"ID":"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf","Type":"ContainerDied","Data":"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c"} Sep 14 01:59:07.075619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.075241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55" event={"ID":"7c44ce5d-4f0a-40d1-8f98-dadee2624fbf","Type":"ContainerDied","Data":"a83257c0942addf97feed1f193bf391fbf35f86180c51e93f611e9b5c11c1a8c"} Sep 14 01:59:07.075619 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.075277 2573 scope.go:117] "RemoveContainer" containerID="0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c" Sep 14 01:59:07.086502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.086484 2573 scope.go:117] "RemoveContainer" containerID="0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c" Sep 14 01:59:07.086773 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:07.086753 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c\": container with ID starting with 0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c not found: ID does not exist" containerID="0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c" Sep 14 01:59:07.086838 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.086785 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c"} err="failed to get container status \"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c\": rpc error: code = NotFound desc = could not find container \"0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c\": container with ID starting with 0cf31a402e4d4584890d78e961fae556baf42a4fb8eec1106564b933c7c1d30c not found: ID does not exist" Sep 14 01:59:07.100611 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.100590 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:59:07.104838 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.104818 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-99c616-5f59f8cf47-9jf55"] Sep 14 01:59:07.863335 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.863302 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" path="/var/lib/kubelet/pods/7c44ce5d-4f0a-40d1-8f98-dadee2624fbf/volumes" Sep 14 01:59:07.977187 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.977156 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:59:07.977463 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:07.977437 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" podUID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" containerName="payload-processing" containerID="cri-o://17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1" gracePeriod=30 Sep 14 01:59:08.261381 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.261354 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:59:08.381377 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.381346 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume\") pod \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " Sep 14 01:59:08.381560 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.381447 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcqhn\" (UniqueName: \"kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn\") pod \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\" (UID: \"40995596-4214-4c7c-a88c-d8a1a7cd2ac1\") " Sep 14 01:59:08.381776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.381741 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40995596-4214-4c7c-a88c-d8a1a7cd2ac1" (UID: "40995596-4214-4c7c-a88c-d8a1a7cd2ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:08.383778 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.383758 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn" (OuterVolumeSpecName: "kube-api-access-bcqhn") pod "40995596-4214-4c7c-a88c-d8a1a7cd2ac1" (UID: "40995596-4214-4c7c-a88c-d8a1a7cd2ac1"). InnerVolumeSpecName "kube-api-access-bcqhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:08.482871 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.482795 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqhn\" (UniqueName: \"kubernetes.io/projected/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-kube-api-access-bcqhn\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:08.482871 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:08.482822 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40995596-4214-4c7c-a88c-d8a1a7cd2ac1-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:09.089161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.089122 2573 generic.go:356] "Generic (PLEG): container finished" podID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" containerID="17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1" exitCode=0 Sep 14 01:59:09.089353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.089173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" event={"ID":"40995596-4214-4c7c-a88c-d8a1a7cd2ac1","Type":"ContainerDied","Data":"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1"} Sep 14 01:59:09.089353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.089190 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" Sep 14 01:59:09.089353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.089225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7" event={"ID":"40995596-4214-4c7c-a88c-d8a1a7cd2ac1","Type":"ContainerDied","Data":"69ec3b2c271bbc46256a65fc4b0635d97dda7f9f3e6ddf42584d0b8407e4f47a"} Sep 14 01:59:09.089353 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.089260 2573 scope.go:117] "RemoveContainer" containerID="17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1" Sep 14 01:59:09.098884 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.098721 2573 scope.go:117] "RemoveContainer" containerID="17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1" Sep 14 01:59:09.099071 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:09.099051 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1\": container with ID starting with 17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1 not found: ID does not exist" containerID="17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1" Sep 14 01:59:09.099161 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.099080 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1"} err="failed to get container status \"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1\": rpc error: code = NotFound desc = could not find container \"17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1\": container with ID starting with 17976641e5bb4d4836ccf54791351870d893970046539e8e26bc8de9f5435ab1 not found: ID does not exist" Sep 14 01:59:09.113792 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.113767 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:59:09.120208 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.120185 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-318a3c-6cc85d96bc-kskn7"] Sep 14 01:59:09.579219 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.579151 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" podUID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" containerName="main" containerID="cri-o://b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28" gracePeriod=45 Sep 14 01:59:09.585364 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.585328 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" podUID="4fe5518e-1519-483e-861b-0599a0aff40d" containerName="main" containerID="cri-o://184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a" gracePeriod=45 Sep 14 01:59:09.836183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.836118 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:59:09.868351 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.868317 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" path="/var/lib/kubelet/pods/40995596-4214-4c7c-a88c-d8a1a7cd2ac1/volumes" Sep 14 01:59:09.873786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.873764 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:59:09.897500 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897467 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897648 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897548 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897648 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897598 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897685 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897776 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897717 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8q9d\" (UniqueName: \"kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897871 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897784 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache\") pod \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\" (UID: \"0465dd4d-0dfe-4768-96ca-1a76405f5f90\") " Sep 14 01:59:09.897871 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897792 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.897979 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.897959 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home" (OuterVolumeSpecName: "home") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.898145 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.898099 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache" (OuterVolumeSpecName: "model-cache") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.898145 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.898126 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tmp-dir\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:09.898330 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.898147 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-home\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:09.900525 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.900495 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:59:09.900677 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.900650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm" (OuterVolumeSpecName: "dshm") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.900809 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.900721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d" (OuterVolumeSpecName: "kube-api-access-z8q9d") pod "0465dd4d-0dfe-4768-96ca-1a76405f5f90" (UID: "0465dd4d-0dfe-4768-96ca-1a76405f5f90"). InnerVolumeSpecName "kube-api-access-z8q9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:09.999467 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999433 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999512 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999614 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999572 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999605 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999676 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz8jg\" (UniqueName: \"kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm\") pod \"4fe5518e-1519-483e-861b-0599a0aff40d\" (UID: \"4fe5518e-1519-483e-861b-0599a0aff40d\") " Sep 14 01:59:09.999753 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999721 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.999949 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999831 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:09.999949 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:09.999907 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home" (OuterVolumeSpecName: "home") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:10.000173 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000143 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-model-cache\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000173 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000171 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-model-cache\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000185 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0465dd4d-0dfe-4768-96ca-1a76405f5f90-dshm\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000206 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-home\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000215 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0465dd4d-0dfe-4768-96ca-1a76405f5f90-tls-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000222 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-tmp-dir\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.000396 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.000231 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8q9d\" (UniqueName: \"kubernetes.io/projected/0465dd4d-0dfe-4768-96ca-1a76405f5f90-kube-api-access-z8q9d\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.001927 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.001910 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg" (OuterVolumeSpecName: "kube-api-access-lz8jg") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "kube-api-access-lz8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:10.002403 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.002385 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 01:59:10.002463 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.002426 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm" (OuterVolumeSpecName: "dshm") pod "4fe5518e-1519-483e-861b-0599a0aff40d" (UID: "4fe5518e-1519-483e-861b-0599a0aff40d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:10.097343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.097228 2573 generic.go:356] "Generic (PLEG): container finished" podID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" containerID="b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28" exitCode=0 Sep 14 01:59:10.097343 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.097304 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" Sep 14 01:59:10.097535 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.097300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" event={"ID":"0465dd4d-0dfe-4768-96ca-1a76405f5f90","Type":"ContainerDied","Data":"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28"} Sep 14 01:59:10.097535 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.097416 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9" event={"ID":"0465dd4d-0dfe-4768-96ca-1a76405f5f90","Type":"ContainerDied","Data":"76b8281a2324ae2d7f5ccbe636914e426b411b997208a9ac52dd9d103b2538c2"} Sep 14 01:59:10.097535 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.097442 2573 scope.go:117] "RemoveContainer" containerID="b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28" Sep 14 01:59:10.098781 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.098757 2573 generic.go:356] "Generic (PLEG): container finished" podID="4fe5518e-1519-483e-861b-0599a0aff40d" containerID="184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a" exitCode=0 Sep 14 01:59:10.098892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.098794 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" Sep 14 01:59:10.098892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.098827 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" event={"ID":"4fe5518e-1519-483e-861b-0599a0aff40d","Type":"ContainerDied","Data":"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a"} Sep 14 01:59:10.098892 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.098858 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k" event={"ID":"4fe5518e-1519-483e-861b-0599a0aff40d","Type":"ContainerDied","Data":"268eb63987ade1e4b982fc9ce0ac6b78c1bd564a9f7e9b01f1fc3332d1a8adb6"} Sep 14 01:59:10.100898 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.100875 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe5518e-1519-483e-861b-0599a0aff40d-tls-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.101017 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.100903 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8jg\" (UniqueName: \"kubernetes.io/projected/4fe5518e-1519-483e-861b-0599a0aff40d-kube-api-access-lz8jg\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.101017 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.100919 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe5518e-1519-483e-861b-0599a0aff40d-dshm\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:10.110090 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.109692 2573 scope.go:117] "RemoveContainer" containerID="b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28" Sep 14 01:59:10.110559 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:10.110453 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28\": container with ID starting with b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28 not found: ID does not exist" containerID="b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28" Sep 14 01:59:10.110559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.110484 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28"} err="failed to get container status \"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28\": rpc error: code = NotFound desc = could not find container \"b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28\": container with ID starting with b5885a87209a5c222557c23545d3988efa8cb0440d017ddd445c81641b365c28 not found: ID does not exist" Sep 14 01:59:10.110559 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.110506 2573 scope.go:117] "RemoveContainer" containerID="184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a" Sep 14 01:59:10.119154 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.119137 2573 scope.go:117] "RemoveContainer" containerID="184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a" Sep 14 01:59:10.119460 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:10.119443 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a\": container with ID starting with 184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a not found: ID does not exist" containerID="184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a" Sep 14 01:59:10.119537 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.119468 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a"} err="failed to get container status \"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a\": rpc error: code = NotFound desc = could not find container \"184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a\": container with ID starting with 184dbfaee4e76a6ee663c94afebb81f34fcea83e6c3b8d13e50bca612931e27a not found: ID does not exist" Sep 14 01:59:10.123530 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.123508 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:59:10.130890 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.130869 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/distinct-w0-99c616-kserve-d8849655f-c5sf9"] Sep 14 01:59:10.145955 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.145925 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:59:10.146075 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:10.145973 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-99c616/embedding-w0-99c616-kserve-7b5f6c55fb-2nr8k"] Sep 14 01:59:11.861708 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:11.861671 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" path="/var/lib/kubelet/pods/0465dd4d-0dfe-4768-96ca-1a76405f5f90/volumes" Sep 14 01:59:11.862285 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:11.862244 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe5518e-1519-483e-861b-0599a0aff40d" path="/var/lib/kubelet/pods/4fe5518e-1519-483e-861b-0599a0aff40d/volumes" Sep 14 01:59:25.353994 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.353960 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:59:25.354553 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.354302 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" podUID="487df5c2-8f99-4197-a2a5-372291012fae" containerName="payload-pre-processing" containerID="cri-o://352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259" gracePeriod=30 Sep 14 01:59:25.644346 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.644319 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:59:25.761717 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.761682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp\") pod \"487df5c2-8f99-4197-a2a5-372291012fae\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " Sep 14 01:59:25.761919 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.761736 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume\") pod \"487df5c2-8f99-4197-a2a5-372291012fae\" (UID: \"487df5c2-8f99-4197-a2a5-372291012fae\") " Sep 14 01:59:25.762380 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.762349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "487df5c2-8f99-4197-a2a5-372291012fae" (UID: "487df5c2-8f99-4197-a2a5-372291012fae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:25.764288 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.764221 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp" (OuterVolumeSpecName: "kube-api-access-4bwbp") pod "487df5c2-8f99-4197-a2a5-372291012fae" (UID: "487df5c2-8f99-4197-a2a5-372291012fae"). InnerVolumeSpecName "kube-api-access-4bwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:25.863000 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.862965 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/487df5c2-8f99-4197-a2a5-372291012fae-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:25.863000 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:25.863000 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bwbp\" (UniqueName: \"kubernetes.io/projected/487df5c2-8f99-4197-a2a5-372291012fae-kube-api-access-4bwbp\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:26.189197 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.189162 2573 generic.go:356] "Generic (PLEG): container finished" podID="487df5c2-8f99-4197-a2a5-372291012fae" containerID="352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259" exitCode=0 Sep 14 01:59:26.189410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.189209 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" Sep 14 01:59:26.189410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.189246 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" event={"ID":"487df5c2-8f99-4197-a2a5-372291012fae","Type":"ContainerDied","Data":"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259"} Sep 14 01:59:26.189410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.189318 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw" event={"ID":"487df5c2-8f99-4197-a2a5-372291012fae","Type":"ContainerDied","Data":"bc4a898558f3f4c27c84378b36044906548a010f35872d09be552c62122eafa1"} Sep 14 01:59:26.189410 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.189342 2573 scope.go:117] "RemoveContainer" containerID="352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259" Sep 14 01:59:26.198287 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.198263 2573 scope.go:117] "RemoveContainer" containerID="352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259" Sep 14 01:59:26.198589 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:26.198560 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259\": container with ID starting with 352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259 not found: ID does not exist" containerID="352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259" Sep 14 01:59:26.198681 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.198596 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259"} err="failed to get container status \"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259\": rpc error: code = NotFound desc = could not find container \"352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259\": container with ID starting with 352af4516a2ea9013a4d8f11d93ca6731b2ce2a598b33ba8672a472453d14259 not found: ID does not exist" Sep 14 01:59:26.210613 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.210581 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:59:26.214480 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:26.214460 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-38e192-76554f8c54-gsjhw"] Sep 14 01:59:27.862786 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:27.862751 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487df5c2-8f99-4197-a2a5-372291012fae" path="/var/lib/kubelet/pods/487df5c2-8f99-4197-a2a5-372291012fae/volumes" Sep 14 01:59:32.599117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:32.599080 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:59:32.601594 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:32.599330 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" containerID="cri-o://fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541" gracePeriod=30 Sep 14 01:59:37.864716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.864653 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:59:37.983336 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983307 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983336 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983339 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983374 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983399 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983430 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983514 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983569 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983542 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983576 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d2vg\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg\") pod \"c217e024-389a-4e96-a054-2edf4df559f7\" (UID: \"c217e024-389a-4e96-a054-2edf4df559f7\") " Sep 14 01:59:37.983881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983628 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:37.983881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983646 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:37.983881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983803 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:37.983881 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983866 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:37.984117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983978 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:37.984117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.983998 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:37.984117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.984013 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:37.984117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.984026 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:37.984117 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.984043 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:37.985680 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.985648 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:37.985939 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.985909 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg" (OuterVolumeSpecName: "kube-api-access-2d2vg") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "kube-api-access-2d2vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:37.986008 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.985965 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:59:37.986301 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:37.986272 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "c217e024-389a-4e96-a054-2edf4df559f7" (UID: "c217e024-389a-4e96-a054-2edf4df559f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:38.085529 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.085498 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c217e024-389a-4e96-a054-2edf4df559f7-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:38.085529 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.085525 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:38.085529 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.085534 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c217e024-389a-4e96-a054-2edf4df559f7-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:38.085742 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.085542 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d2vg\" (UniqueName: \"kubernetes.io/projected/c217e024-389a-4e96-a054-2edf4df559f7-kube-api-access-2d2vg\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:38.085742 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.085551 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c217e024-389a-4e96-a054-2edf4df559f7-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:38.255654 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.255624 2573 generic.go:356] "Generic (PLEG): container finished" podID="c217e024-389a-4e96-a054-2edf4df559f7" containerID="fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541" exitCode=0 Sep 14 01:59:38.255823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.255671 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" Sep 14 01:59:38.255823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.255716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerDied","Data":"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541"} Sep 14 01:59:38.255823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.255769 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx" event={"ID":"c217e024-389a-4e96-a054-2edf4df559f7","Type":"ContainerDied","Data":"2c9d1e09a8d9b8981defc9ba4342e54963358425ebe4fcc66aac6d450d63bfc8"} Sep 14 01:59:38.255823 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.255791 2573 scope.go:117] "RemoveContainer" containerID="fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541" Sep 14 01:59:38.265545 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.265527 2573 scope.go:117] "RemoveContainer" containerID="089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13" Sep 14 01:59:38.273871 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.273853 2573 scope.go:117] "RemoveContainer" containerID="fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541" Sep 14 01:59:38.274129 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:38.274111 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541\": container with ID starting with fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541 not found: ID does not exist" containerID="fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541" Sep 14 01:59:38.274179 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.274138 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541"} err="failed to get container status \"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541\": rpc error: code = NotFound desc = could not find container \"fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541\": container with ID starting with fa6d23c2cdb9a1920b79002342afb39d24741c2ac970c905a208bebd282ac541 not found: ID does not exist" Sep 14 01:59:38.274179 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.274154 2573 scope.go:117] "RemoveContainer" containerID="089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13" Sep 14 01:59:38.274466 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:38.274447 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13\": container with ID starting with 089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13 not found: ID does not exist" containerID="089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13" Sep 14 01:59:38.274518 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.274472 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13"} err="failed to get container status \"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13\": rpc error: code = NotFound desc = could not find container \"089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13\": container with ID starting with 089479d865b5662dea7da4c9fa74e33cf2654992cf5446346ccc343d283dec13 not found: ID does not exist" Sep 14 01:59:38.279510 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.279488 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:59:38.283651 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:38.283627 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-38e192-openshift-default-656755d6f4-ffzjx"] Sep 14 01:59:39.861226 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:39.861196 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c217e024-389a-4e96-a054-2edf4df559f7" path="/var/lib/kubelet/pods/c217e024-389a-4e96-a054-2edf4df559f7/volumes" Sep 14 01:59:43.407905 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.407868 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:59:43.408378 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.408177 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" podUID="88eac46a-6714-44ca-bfa7-7876b08f2214" containerName="payload-processing" containerID="cri-o://4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969" gracePeriod=30 Sep 14 01:59:43.685535 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.685506 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:59:43.743326 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.743288 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kz6l\" (UniqueName: \"kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l\") pod \"88eac46a-6714-44ca-bfa7-7876b08f2214\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " Sep 14 01:59:43.743513 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.743336 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume\") pod \"88eac46a-6714-44ca-bfa7-7876b08f2214\" (UID: \"88eac46a-6714-44ca-bfa7-7876b08f2214\") " Sep 14 01:59:43.743708 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.743670 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88eac46a-6714-44ca-bfa7-7876b08f2214" (UID: "88eac46a-6714-44ca-bfa7-7876b08f2214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:43.745581 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.745560 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l" (OuterVolumeSpecName: "kube-api-access-8kz6l") pod "88eac46a-6714-44ca-bfa7-7876b08f2214" (UID: "88eac46a-6714-44ca-bfa7-7876b08f2214"). InnerVolumeSpecName "kube-api-access-8kz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:43.844398 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.844365 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kz6l\" (UniqueName: \"kubernetes.io/projected/88eac46a-6714-44ca-bfa7-7876b08f2214-kube-api-access-8kz6l\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:43.844398 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:43.844395 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88eac46a-6714-44ca-bfa7-7876b08f2214-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:44.290586 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.290545 2573 generic.go:356] "Generic (PLEG): container finished" podID="88eac46a-6714-44ca-bfa7-7876b08f2214" containerID="4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969" exitCode=0 Sep 14 01:59:44.290586 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.290591 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" Sep 14 01:59:44.290827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.290621 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" event={"ID":"88eac46a-6714-44ca-bfa7-7876b08f2214","Type":"ContainerDied","Data":"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969"} Sep 14 01:59:44.290827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.290664 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj" event={"ID":"88eac46a-6714-44ca-bfa7-7876b08f2214","Type":"ContainerDied","Data":"295588f5d7f55126f0b8313bbb71380f13ca4264c969bfc26d927baaff6716f6"} Sep 14 01:59:44.290827 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.290694 2573 scope.go:117] "RemoveContainer" containerID="4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969" Sep 14 01:59:44.300999 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.300968 2573 scope.go:117] "RemoveContainer" containerID="4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969" Sep 14 01:59:44.301244 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:44.301219 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969\": container with ID starting with 4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969 not found: ID does not exist" containerID="4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969" Sep 14 01:59:44.301373 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.301275 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969"} err="failed to get container status \"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969\": rpc error: code = NotFound desc = could not find container \"4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969\": container with ID starting with 4cf8749472efe8a40df4a5ef500a72195b667785aa0fa15a119b945fcc927969 not found: ID does not exist" Sep 14 01:59:44.307660 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.307633 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:59:44.309502 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:44.309481 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-7ff93e-656467bb9f-d7wfj"] Sep 14 01:59:45.864412 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:45.864368 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88eac46a-6714-44ca-bfa7-7876b08f2214" path="/var/lib/kubelet/pods/88eac46a-6714-44ca-bfa7-7876b08f2214/volumes" Sep 14 01:59:49.980227 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:49.980194 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:59:49.980734 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:49.980438 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" containerID="cri-o://f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9" gracePeriod=30 Sep 14 01:59:52.328840 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:52.328800 2573 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 01:59:52.329244 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:52.328872 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 01:59:54.016769 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:54.016738 2573 scope.go:117] "RemoveContainer" containerID="5830055cb7d0dee513ab270d0209ec00c773d5b0445c4ec7bfadaf8059aaecab" Sep 14 01:59:55.239914 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.239892 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:59:55.350924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.350834 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.350924 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.350870 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.350931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.350964 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351159 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351082 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351162 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:55.351358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351175 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5dkb\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351218 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351271 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351358 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351307 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket\") pod \"72585742-6b07-4546-9a88-b4da4c187214\" (UID: \"72585742-6b07-4546-9a88-b4da4c187214\") " Sep 14 01:59:55.351611 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351461 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data" (OuterVolumeSpecName: "istio-data") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:55.351682 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351648 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 01:59:55.351965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351890 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.351965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351915 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72585742-6b07-4546-9a88-b4da4c187214-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.351965 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.351930 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.352187 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352111 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:55.352187 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352132 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:55.352711 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352683 2573 generic.go:356] "Generic (PLEG): container finished" podID="72585742-6b07-4546-9a88-b4da4c187214" containerID="f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9" exitCode=0 Sep 14 01:59:55.352818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352751 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" Sep 14 01:59:55.352818 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352775 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerDied","Data":"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9"} Sep 14 01:59:55.352932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352843 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4" event={"ID":"72585742-6b07-4546-9a88-b4da4c187214","Type":"ContainerDied","Data":"6702433ef18ae305524e3c8427e9440af4d7aa5a33a9bebcddaf5d8aef8389f4"} Sep 14 01:59:55.352932 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.352878 2573 scope.go:117] "RemoveContainer" containerID="f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9" Sep 14 01:59:55.353716 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.353650 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token" (OuterVolumeSpecName: "istio-token") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:55.353868 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.353715 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 01:59:55.353868 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.353747 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 01:59:55.353868 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.353834 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb" (OuterVolumeSpecName: "kube-api-access-h5dkb") pod "72585742-6b07-4546-9a88-b4da4c187214" (UID: "72585742-6b07-4546-9a88-b4da4c187214"). InnerVolumeSpecName "kube-api-access-h5dkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 01:59:55.371979 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.371958 2573 scope.go:117] "RemoveContainer" containerID="ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0" Sep 14 01:59:55.379863 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.379845 2573 scope.go:117] "RemoveContainer" containerID="f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9" Sep 14 01:59:55.380140 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:55.380124 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9\": container with ID starting with f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9 not found: ID does not exist" containerID="f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9" Sep 14 01:59:55.380183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.380149 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9"} err="failed to get container status \"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9\": rpc error: code = NotFound desc = could not find container \"f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9\": container with ID starting with f8bbdb4078a0bf9536afe04d29168019b11404a0f1b29314254ed27fcac54fe9 not found: ID does not exist" Sep 14 01:59:55.380183 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.380166 2573 scope.go:117] "RemoveContainer" containerID="ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0" Sep 14 01:59:55.380432 ip-10-0-132-205 kubenswrapper[2573]: E0914 01:59:55.380414 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0\": container with ID starting with ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0 not found: ID does not exist" containerID="ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0" Sep 14 01:59:55.380478 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.380439 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0"} err="failed to get container status \"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0\": rpc error: code = NotFound desc = could not find container \"ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0\": container with ID starting with ed3ac1dc20e4da3943396683c5db77b8b08f7ff9f9e9c4c6cf7f959bc8042cb0 not found: ID does not exist" Sep 14 01:59:55.452992 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.452963 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.452992 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.452988 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dkb\" (UniqueName: \"kubernetes.io/projected/72585742-6b07-4546-9a88-b4da4c187214-kube-api-access-h5dkb\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.453134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.453000 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.453134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.453009 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.453134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.453019 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72585742-6b07-4546-9a88-b4da4c187214-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.453134 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.453026 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72585742-6b07-4546-9a88-b4da4c187214-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 01:59:55.675119 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.675089 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:59:55.679149 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.679122 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7ff93e-openshift-default-7dd6bc6656-k8ns4"] Sep 14 01:59:55.862661 ip-10-0-132-205 kubenswrapper[2573]: I0914 01:59:55.862625 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72585742-6b07-4546-9a88-b4da4c187214" path="/var/lib/kubelet/pods/72585742-6b07-4546-9a88-b4da4c187214/volumes" Sep 14 02:00:11.270194 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.270158 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 02:00:11.271114 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.271053 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" podUID="3b94e090-72be-4b9b-9265-0cee35429cd4" containerName="maas-api" containerID="cri-o://ded63bde2c0a7545b687a6e3809237b61079a1df333c2c0d4c78df2b50aaf7cf" gracePeriod=30 Sep 14 02:00:11.434233 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.434198 2573 generic.go:356] "Generic (PLEG): container finished" podID="3b94e090-72be-4b9b-9265-0cee35429cd4" containerID="ded63bde2c0a7545b687a6e3809237b61079a1df333c2c0d4c78df2b50aaf7cf" exitCode=0 Sep 14 02:00:11.434480 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.434285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" event={"ID":"3b94e090-72be-4b9b-9265-0cee35429cd4","Type":"ContainerDied","Data":"ded63bde2c0a7545b687a6e3809237b61079a1df333c2c0d4c78df2b50aaf7cf"} Sep 14 02:00:11.528307 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.528234 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 02:00:11.600852 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.600824 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs\") pod \"3b94e090-72be-4b9b-9265-0cee35429cd4\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " Sep 14 02:00:11.600852 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.600863 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") pod \"3b94e090-72be-4b9b-9265-0cee35429cd4\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " Sep 14 02:00:11.601116 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.600891 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnkws\" (UniqueName: \"kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws\") pod \"3b94e090-72be-4b9b-9265-0cee35429cd4\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " Sep 14 02:00:11.601116 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.600989 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca\") pod \"3b94e090-72be-4b9b-9265-0cee35429cd4\" (UID: \"3b94e090-72be-4b9b-9265-0cee35429cd4\") " Sep 14 02:00:11.601379 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.601349 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b94e090-72be-4b9b-9265-0cee35429cd4" (UID: "3b94e090-72be-4b9b-9265-0cee35429cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:00:11.603162 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.603142 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b94e090-72be-4b9b-9265-0cee35429cd4" (UID: "3b94e090-72be-4b9b-9265-0cee35429cd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:00:11.603265 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.603216 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws" (OuterVolumeSpecName: "kube-api-access-dnkws") pod "3b94e090-72be-4b9b-9265-0cee35429cd4" (UID: "3b94e090-72be-4b9b-9265-0cee35429cd4"). InnerVolumeSpecName "kube-api-access-dnkws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:11.603310 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.603245 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b94e090-72be-4b9b-9265-0cee35429cd4" (UID: "3b94e090-72be-4b9b-9265-0cee35429cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:00:11.702227 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.702195 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.702227 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.702224 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b94e090-72be-4b9b-9265-0cee35429cd4-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.702227 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.702233 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkws\" (UniqueName: \"kubernetes.io/projected/3b94e090-72be-4b9b-9265-0cee35429cd4-kube-api-access-dnkws\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:00:11.702481 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:11.702243 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b94e090-72be-4b9b-9265-0cee35429cd4-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:00:12.440397 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:12.440368 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" Sep 14 02:00:12.440793 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:12.440365 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6" event={"ID":"3b94e090-72be-4b9b-9265-0cee35429cd4","Type":"ContainerDied","Data":"eae239f5c5ef631e47b07fd4ecc71e7c0f2edff1a4db12f965a5f2dbf2031a6a"} Sep 14 02:00:12.440793 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:12.440542 2573 scope.go:117] "RemoveContainer" containerID="ded63bde2c0a7545b687a6e3809237b61079a1df333c2c0d4c78df2b50aaf7cf" Sep 14 02:00:12.456591 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:12.456566 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 02:00:12.459604 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:12.459583 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8e5935-646fd4c8dd-jhzw6"] Sep 14 02:00:13.862954 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:13.862922 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b94e090-72be-4b9b-9265-0cee35429cd4" path="/var/lib/kubelet/pods/3b94e090-72be-4b9b-9265-0cee35429cd4/volumes" Sep 14 02:00:15.292635 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.292603 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7446-74ce9f"] Sep 14 02:00:15.292998 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.292982 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9e9490-d578-4bda-8f9e-8bba00976835" containerName="payload-processing" Sep 14 02:00:15.292998 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.292993 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9e9490-d578-4bda-8f9e-8bba00976835" containerName="payload-processing" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293005 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293011 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293024 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293030 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293036 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" containerName="main" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293040 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" containerName="main" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293048 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" containerName="payload-pre-processing" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293053 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" containerName="payload-pre-processing" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293064 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b94e090-72be-4b9b-9265-0cee35429cd4" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293068 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b94e090-72be-4b9b-9265-0cee35429cd4" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293075 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293081 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" containerName="maas-api" Sep 14 02:00:15.293084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293088 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293093 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293102 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293107 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293113 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487df5c2-8f99-4197-a2a5-372291012fae" containerName="payload-pre-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293118 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="487df5c2-8f99-4197-a2a5-372291012fae" containerName="payload-pre-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293124 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88eac46a-6714-44ca-bfa7-7876b08f2214" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293129 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="88eac46a-6714-44ca-bfa7-7876b08f2214" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293139 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe5518e-1519-483e-861b-0599a0aff40d" containerName="main" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293144 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe5518e-1519-483e-861b-0599a0aff40d" containerName="main" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293213 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293223 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c44ce5d-4f0a-40d1-8f98-dadee2624fbf" containerName="maas-api" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293230 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="88eac46a-6714-44ca-bfa7-7876b08f2214" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293240 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9e9490-d578-4bda-8f9e-8bba00976835" containerName="payload-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293265 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b94e090-72be-4b9b-9265-0cee35429cd4" containerName="maas-api" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293271 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="487df5c2-8f99-4197-a2a5-372291012fae" containerName="payload-pre-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293277 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293284 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe5518e-1519-483e-861b-0599a0aff40d" containerName="main" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293289 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cf1cb4e-947c-48de-928f-341c64fa3ddf" containerName="maas-api" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293294 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="449c7d04-8a8a-4706-b873-f7eaf13e8dbd" containerName="payload-pre-processing" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293299 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293304 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0465dd4d-0dfe-4768-96ca-1a76405f5f90" containerName="main" Sep 14 02:00:15.293488 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.293309 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40995596-4214-4c7c-a88c-d8a1a7cd2ac1" containerName="payload-processing" Sep 14 02:00:15.297845 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.297823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:15.300553 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.300523 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-74qx8\"" Sep 14 02:00:15.301867 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.301842 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-74ce9f"] Sep 14 02:00:15.334094 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.334059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77j7h\" (UniqueName: \"kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h\") pod \"test-curl-7446-74ce9f\" (UID: \"649e4c37-86a0-4452-8fff-a471207fbd06\") " pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:15.435391 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.435361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77j7h\" (UniqueName: \"kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h\") pod \"test-curl-7446-74ce9f\" (UID: \"649e4c37-86a0-4452-8fff-a471207fbd06\") " pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:15.443122 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.443101 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77j7h\" (UniqueName: \"kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h\") pod \"test-curl-7446-74ce9f\" (UID: \"649e4c37-86a0-4452-8fff-a471207fbd06\") " pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:15.609033 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.608952 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:15.881910 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.881884 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-74ce9f"] Sep 14 02:00:15.883527 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:00:15.883494 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod649e4c37_86a0_4452_8fff_a471207fbd06.slice/crio-bb0d5dc7424bacc51042ecc20717375856c6071112a0781f2970dd44c944e4c1 WatchSource:0}: Error finding container bb0d5dc7424bacc51042ecc20717375856c6071112a0781f2970dd44c944e4c1: Status 404 returned error can't find the container with id bb0d5dc7424bacc51042ecc20717375856c6071112a0781f2970dd44c944e4c1 Sep 14 02:00:15.885550 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:15.885529 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 02:00:16.462233 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:16.462185 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-74ce9f" event={"ID":"649e4c37-86a0-4452-8fff-a471207fbd06","Type":"ContainerStarted","Data":"bb0d5dc7424bacc51042ecc20717375856c6071112a0781f2970dd44c944e4c1"} Sep 14 02:00:19.487343 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:19.487298 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-74ce9f" event={"ID":"649e4c37-86a0-4452-8fff-a471207fbd06","Type":"ContainerStarted","Data":"bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55"} Sep 14 02:00:19.500192 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:19.500145 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7446-74ce9f" podStartSLOduration=1.702966744 podStartE2EDuration="4.500128014s" totalImagesPullingTime="2.79716127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:00:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:00:19.499854833 +0000 UTC m=+1766.172971008" watchObservedRunningTime="2026-09-14 02:00:19.500128014 +0000 UTC m=+1766.173244191" Sep 14 02:00:20.088507 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:20.088468 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7446-74ce9f"] Sep 14 02:00:20.092508 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:20.092481 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7446-74ce9f"] Sep 14 02:00:20.492205 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:00:20.492170 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7446-74ce9f" container="test-curl-7446-74ce9f" Sep 14 02:00:21.496351 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:21.496309 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7446-74ce9f" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" containerName="test-curl-7446-74ce9f" containerID="cri-o://bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55" gracePeriod=2 Sep 14 02:00:21.499094 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:00:21.499061 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7446-74ce9f\" is forbidden: User \"system:node:ip-10-0-132-205.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-205.ec2.internal' and this object" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:23.631385 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:23.631356 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:23.717100 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:23.717069 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77j7h\" (UniqueName: \"kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h\") pod \"649e4c37-86a0-4452-8fff-a471207fbd06\" (UID: \"649e4c37-86a0-4452-8fff-a471207fbd06\") " Sep 14 02:00:23.719234 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:23.719211 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h" (OuterVolumeSpecName: "kube-api-access-77j7h") pod "649e4c37-86a0-4452-8fff-a471207fbd06" (UID: "649e4c37-86a0-4452-8fff-a471207fbd06"). InnerVolumeSpecName "kube-api-access-77j7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:00:23.818569 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:23.818491 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77j7h\" (UniqueName: \"kubernetes.io/projected/649e4c37-86a0-4452-8fff-a471207fbd06-kube-api-access-77j7h\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:00:23.861381 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:23.861352 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" path="/var/lib/kubelet/pods/649e4c37-86a0-4452-8fff-a471207fbd06/volumes" Sep 14 02:00:24.512721 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:24.512685 2573 generic.go:356] "Generic (PLEG): container finished" podID="649e4c37-86a0-4452-8fff-a471207fbd06" containerID="bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55" exitCode=137 Sep 14 02:00:24.512922 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:24.512744 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-74ce9f" Sep 14 02:00:24.512922 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:24.512792 2573 scope.go:117] "RemoveContainer" containerID="bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55" Sep 14 02:00:24.521416 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:24.521313 2573 scope.go:117] "RemoveContainer" containerID="bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55" Sep 14 02:00:24.521608 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:00:24.521590 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55\": container with ID starting with bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55 not found: ID does not exist" containerID="bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55" Sep 14 02:00:24.521659 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:24.521622 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55"} err="failed to get container status \"bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55\": rpc error: code = NotFound desc = could not find container \"bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55\": container with ID starting with bdbc75c763826f9f2fde964e2c90972e0a7a527b776c42aa63350a4f2d2f6a55 not found: ID does not exist" Sep 14 02:00:29.311190 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311155 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311538 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311551 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311564 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" containerName="test-curl-7446-74ce9f" Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311571 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" containerName="test-curl-7446-74ce9f" Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311587 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:29.311662 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311594 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="72585742-6b07-4546-9a88-b4da4c187214" containerName="istio-proxy" Sep 14 02:00:29.312137 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311672 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c217e024-389a-4e96-a054-2edf4df559f7" containerName="istio-proxy" Sep 14 02:00:29.312137 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.311686 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="649e4c37-86a0-4452-8fff-a471207fbd06" containerName="test-curl-7446-74ce9f" Sep 14 02:00:29.316433 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.316410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.319715 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.319692 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d3ac7\"/\"sub-test-model-5d3ac7-kserve-self-signed-certs\"" Sep 14 02:00:29.319919 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.319750 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d3ac7\"/\"kube-root-ca.crt\"" Sep 14 02:00:29.319919 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.319750 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d3ac7\"/\"openshift-service-ca.crt\"" Sep 14 02:00:29.320269 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.319767 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-5d3ac7\"/\"default-dockercfg-gw7rs\"" Sep 14 02:00:29.323316 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.323295 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:00:29.370411 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370384 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.370556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9qb\" (UniqueName: \"kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.370556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.370556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370466 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.370556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370514 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.370706 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.370569 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471785 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471961 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471961 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9qb\" (UniqueName: \"kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471961 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471961 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.471961 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.471911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.472240 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.472211 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.472341 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.472280 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.472386 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.472332 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.474241 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.474219 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.474434 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.474416 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.478851 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.478830 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9qb\" (UniqueName: \"kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb\") pod \"sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.628693 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.628605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:29.901311 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:29.901284 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:00:29.904036 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:00:29.904005 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb129ac68_f177_464d_8445_004488912371.slice/crio-f96561d44346650752b5b65a744978d3681e3cbc1b7a633bea18edaf84b38b04 WatchSource:0}: Error finding container f96561d44346650752b5b65a744978d3681e3cbc1b7a633bea18edaf84b38b04: Status 404 returned error can't find the container with id f96561d44346650752b5b65a744978d3681e3cbc1b7a633bea18edaf84b38b04 Sep 14 02:00:30.545002 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:30.544964 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" event={"ID":"b129ac68-f177-464d-8445-004488912371","Type":"ContainerStarted","Data":"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1"} Sep 14 02:00:30.545382 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:30.545011 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" event={"ID":"b129ac68-f177-464d-8445-004488912371","Type":"ContainerStarted","Data":"f96561d44346650752b5b65a744978d3681e3cbc1b7a633bea18edaf84b38b04"} Sep 14 02:00:30.567486 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:30.567434 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" podStartSLOduration=1.5674187160000002 podStartE2EDuration="1.567418716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:00:30.566550647 +0000 UTC m=+1777.239666822" watchObservedRunningTime="2026-09-14 02:00:30.567418716 +0000 UTC m=+1777.240534891" Sep 14 02:00:39.629587 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:39.629545 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:39.630018 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:39.629610 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:39.642659 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:39.642634 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:00:40.605908 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:00:40.605880 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:01:39.493101 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.493063 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7446-593ba5"] Sep 14 02:01:39.497755 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.497736 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:39.500304 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.500284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-74qx8\"" Sep 14 02:01:39.501435 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.501409 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-593ba5"] Sep 14 02:01:39.542698 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.542667 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lchr6\" (UniqueName: \"kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6\") pod \"test-curl-7446-593ba5\" (UID: \"c669fb23-1f29-499c-8956-d3e900858179\") " pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:39.643686 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.643659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lchr6\" (UniqueName: \"kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6\") pod \"test-curl-7446-593ba5\" (UID: \"c669fb23-1f29-499c-8956-d3e900858179\") " pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:39.651507 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.651474 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lchr6\" (UniqueName: \"kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6\") pod \"test-curl-7446-593ba5\" (UID: \"c669fb23-1f29-499c-8956-d3e900858179\") " pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:39.808326 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:39.808216 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:40.075369 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:40.075343 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-593ba5"] Sep 14 02:01:40.077416 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:01:40.077378 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc669fb23_1f29_499c_8956_d3e900858179.slice/crio-387d75663b0e0dbeae39b91fbf1562e4b8e5a807256a18e4cc281911d6a85f7e WatchSource:0}: Error finding container 387d75663b0e0dbeae39b91fbf1562e4b8e5a807256a18e4cc281911d6a85f7e: Status 404 returned error can't find the container with id 387d75663b0e0dbeae39b91fbf1562e4b8e5a807256a18e4cc281911d6a85f7e Sep 14 02:01:40.884442 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:40.884405 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-593ba5" event={"ID":"c669fb23-1f29-499c-8956-d3e900858179","Type":"ContainerStarted","Data":"fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932"} Sep 14 02:01:40.884442 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:40.884446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-593ba5" event={"ID":"c669fb23-1f29-499c-8956-d3e900858179","Type":"ContainerStarted","Data":"387d75663b0e0dbeae39b91fbf1562e4b8e5a807256a18e4cc281911d6a85f7e"} Sep 14 02:01:40.897060 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:40.897008 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7446-593ba5" podStartSLOduration=1.896988909 podStartE2EDuration="1.896988909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:01:40.896447322 +0000 UTC m=+1847.569563497" watchObservedRunningTime="2026-09-14 02:01:40.896988909 +0000 UTC m=+1847.570105085" Sep 14 02:01:41.474780 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.474741 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7446-593ba5"] Sep 14 02:01:41.480650 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.480622 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7446-593ba5"] Sep 14 02:01:41.597839 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.597759 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7446-7f215c"] Sep 14 02:01:41.598139 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.598126 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c669fb23-1f29-499c-8956-d3e900858179" containerName="test-curl-7446-593ba5" Sep 14 02:01:41.598193 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.598140 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c669fb23-1f29-499c-8956-d3e900858179" containerName="test-curl-7446-593ba5" Sep 14 02:01:41.598234 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.598199 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c669fb23-1f29-499c-8956-d3e900858179" containerName="test-curl-7446-593ba5" Sep 14 02:01:41.601921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.601896 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:41.604294 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.604265 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-7f215c"] Sep 14 02:01:41.664330 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.664306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw\") pod \"test-curl-7446-7f215c\" (UID: \"d25d9cda-51ad-47e6-b126-083a5808b163\") " pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:41.765796 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.765768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw\") pod \"test-curl-7446-7f215c\" (UID: \"d25d9cda-51ad-47e6-b126-083a5808b163\") " pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:41.773454 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.773435 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw\") pod \"test-curl-7446-7f215c\" (UID: \"d25d9cda-51ad-47e6-b126-083a5808b163\") " pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:41.888707 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:01:41.888679 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7446-593ba5" container="test-curl-7446-593ba5" Sep 14 02:01:41.911906 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:41.911883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:42.183213 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:42.183189 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7446-7f215c"] Sep 14 02:01:42.185046 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:01:42.185020 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd25d9cda_51ad_47e6_b126_083a5808b163.slice/crio-adeae1413cb21141e8d92c2ba71a2ee27b37cf30adf8bc59b308e4f9f1772d00 WatchSource:0}: Error finding container adeae1413cb21141e8d92c2ba71a2ee27b37cf30adf8bc59b308e4f9f1772d00: Status 404 returned error can't find the container with id adeae1413cb21141e8d92c2ba71a2ee27b37cf30adf8bc59b308e4f9f1772d00 Sep 14 02:01:42.894904 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:42.894865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-7f215c" event={"ID":"d25d9cda-51ad-47e6-b126-083a5808b163","Type":"ContainerStarted","Data":"6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4"} Sep 14 02:01:42.894904 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:42.894911 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7446-7f215c" event={"ID":"d25d9cda-51ad-47e6-b126-083a5808b163","Type":"ContainerStarted","Data":"adeae1413cb21141e8d92c2ba71a2ee27b37cf30adf8bc59b308e4f9f1772d00"} Sep 14 02:01:42.895430 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:42.894952 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7446-593ba5" podUID="c669fb23-1f29-499c-8956-d3e900858179" containerName="test-curl-7446-593ba5" containerID="cri-o://fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932" gracePeriod=2 Sep 14 02:01:42.909622 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:42.909576 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7446-7f215c" podStartSLOduration=1.909560863 podStartE2EDuration="1.909560863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:01:42.908059618 +0000 UTC m=+1849.581175793" watchObservedRunningTime="2026-09-14 02:01:42.909560863 +0000 UTC m=+1849.582677099" Sep 14 02:01:43.464324 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:43.464295 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7446-7f215c"] Sep 14 02:01:43.467099 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:43.467071 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7446-7f215c"] Sep 14 02:01:43.899205 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:01:43.899176 2573 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7446-7f215c" container="test-curl-7446-7f215c" Sep 14 02:01:44.904795 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:44.904766 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7446-7f215c" podUID="d25d9cda-51ad-47e6-b126-083a5808b163" containerName="test-curl-7446-7f215c" containerID="cri-o://6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4" gracePeriod=2 Sep 14 02:01:45.074414 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.074391 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:45.198789 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.198709 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lchr6\" (UniqueName: \"kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6\") pod \"c669fb23-1f29-499c-8956-d3e900858179\" (UID: \"c669fb23-1f29-499c-8956-d3e900858179\") " Sep 14 02:01:45.200906 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.200870 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6" (OuterVolumeSpecName: "kube-api-access-lchr6") pod "c669fb23-1f29-499c-8956-d3e900858179" (UID: "c669fb23-1f29-499c-8956-d3e900858179"). InnerVolumeSpecName "kube-api-access-lchr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:01:45.299665 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.299633 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lchr6\" (UniqueName: \"kubernetes.io/projected/c669fb23-1f29-499c-8956-d3e900858179-kube-api-access-lchr6\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:01:45.440729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.440699 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:01:45.862272 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.862221 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c669fb23-1f29-499c-8956-d3e900858179" path="/var/lib/kubelet/pods/c669fb23-1f29-499c-8956-d3e900858179/volumes" Sep 14 02:01:45.910452 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.910424 2573 generic.go:356] "Generic (PLEG): container finished" podID="c669fb23-1f29-499c-8956-d3e900858179" containerID="fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932" exitCode=137 Sep 14 02:01:45.910818 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.910463 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-593ba5" Sep 14 02:01:45.910818 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.910471 2573 scope.go:117] "RemoveContainer" containerID="fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932" Sep 14 02:01:45.919289 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.919266 2573 scope.go:117] "RemoveContainer" containerID="fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932" Sep 14 02:01:45.919598 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:01:45.919570 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932\": container with ID starting with fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932 not found: ID does not exist" containerID="fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932" Sep 14 02:01:45.919679 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:45.919612 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932"} err="failed to get container status \"fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932\": rpc error: code = NotFound desc = could not find container \"fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932\": container with ID starting with fb0738f4c6776231f948c65c10324f0666b1fdf1bb314ef71ba78fdb211f5932 not found: ID does not exist" Sep 14 02:01:47.089187 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.089161 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:47.219314 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.219235 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw\") pod \"d25d9cda-51ad-47e6-b126-083a5808b163\" (UID: \"d25d9cda-51ad-47e6-b126-083a5808b163\") " Sep 14 02:01:47.221520 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.221498 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw" (OuterVolumeSpecName: "kube-api-access-bgnhw") pod "d25d9cda-51ad-47e6-b126-083a5808b163" (UID: "d25d9cda-51ad-47e6-b126-083a5808b163"). InnerVolumeSpecName "kube-api-access-bgnhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:01:47.320220 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.320191 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnhw\" (UniqueName: \"kubernetes.io/projected/d25d9cda-51ad-47e6-b126-083a5808b163-kube-api-access-bgnhw\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:01:47.862297 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.862234 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25d9cda-51ad-47e6-b126-083a5808b163" path="/var/lib/kubelet/pods/d25d9cda-51ad-47e6-b126-083a5808b163/volumes" Sep 14 02:01:47.922449 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.922417 2573 generic.go:356] "Generic (PLEG): container finished" podID="d25d9cda-51ad-47e6-b126-083a5808b163" containerID="6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4" exitCode=137 Sep 14 02:01:47.922645 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.922457 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7446-7f215c" Sep 14 02:01:47.922645 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.922510 2573 scope.go:117] "RemoveContainer" containerID="6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4" Sep 14 02:01:47.931124 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.931105 2573 scope.go:117] "RemoveContainer" containerID="6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4" Sep 14 02:01:47.931423 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:01:47.931405 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4\": container with ID starting with 6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4 not found: ID does not exist" containerID="6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4" Sep 14 02:01:47.931477 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:01:47.931430 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4"} err="failed to get container status \"6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4\": rpc error: code = NotFound desc = could not find container \"6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4\": container with ID starting with 6e3207896c51a77d779c6399ecf3e5d51a7049063b3687afebda0e4cfd4242a4 not found: ID does not exist" Sep 14 02:02:00.460468 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.460398 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" podUID="b129ac68-f177-464d-8445-004488912371" containerName="main" containerID="cri-o://1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1" gracePeriod=45 Sep 14 02:02:00.595742 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.595689 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" podUID="b129ac68-f177-464d-8445-004488912371" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Sep 14 02:02:00.730331 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.730303 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:02:00.844629 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844598 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm9qb\" (UniqueName: \"kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.844794 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844649 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.844794 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844680 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.844885 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844852 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.844946 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844895 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:02:00.844946 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844918 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.844946 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.844944 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home\") pod \"b129ac68-f177-464d-8445-004488912371\" (UID: \"b129ac68-f177-464d-8445-004488912371\") " Sep 14 02:02:00.845160 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.845139 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache" (OuterVolumeSpecName: "model-cache") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:02:00.845247 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.845227 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home" (OuterVolumeSpecName: "home") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:02:00.845367 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.845344 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-home\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.845367 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.845365 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-tmp-dir\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.845367 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.845374 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-model-cache\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.847135 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.847107 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm" (OuterVolumeSpecName: "dshm") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:02:00.847217 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.847193 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:02:00.847288 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.847211 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb" (OuterVolumeSpecName: "kube-api-access-lm9qb") pod "b129ac68-f177-464d-8445-004488912371" (UID: "b129ac68-f177-464d-8445-004488912371"). InnerVolumeSpecName "kube-api-access-lm9qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:02:00.946410 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.946380 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9qb\" (UniqueName: \"kubernetes.io/projected/b129ac68-f177-464d-8445-004488912371-kube-api-access-lm9qb\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.946410 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.946406 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b129ac68-f177-464d-8445-004488912371-tls-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.946410 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.946415 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b129ac68-f177-464d-8445-004488912371-dshm\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:02:00.986359 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.986281 2573 generic.go:356] "Generic (PLEG): container finished" podID="b129ac68-f177-464d-8445-004488912371" containerID="1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1" exitCode=0 Sep 14 02:02:00.986359 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.986340 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" Sep 14 02:02:00.986544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.986345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" event={"ID":"b129ac68-f177-464d-8445-004488912371","Type":"ContainerDied","Data":"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1"} Sep 14 02:02:00.986544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.986404 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p" event={"ID":"b129ac68-f177-464d-8445-004488912371","Type":"ContainerDied","Data":"f96561d44346650752b5b65a744978d3681e3cbc1b7a633bea18edaf84b38b04"} Sep 14 02:02:00.986544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.986429 2573 scope.go:117] "RemoveContainer" containerID="1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1" Sep 14 02:02:00.995459 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.995444 2573 scope.go:117] "RemoveContainer" containerID="1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1" Sep 14 02:02:00.995708 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:02:00.995690 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1\": container with ID starting with 1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1 not found: ID does not exist" containerID="1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1" Sep 14 02:02:00.995759 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:00.995716 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1"} err="failed to get container status \"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1\": rpc error: code = NotFound desc = could not find container \"1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1\": container with ID starting with 1118eaa10716e2eade59154a85e6df570aa47855040b1cdb1a6bfe1f52f9e0a1 not found: ID does not exist" Sep 14 02:02:01.005828 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:01.005806 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:02:01.009709 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:01.009689 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-5d3ac7/sub-test-model-5d3ac7-kserve-74cd77c5b9-jsv6p"] Sep 14 02:02:01.863086 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:02:01.863054 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b129ac68-f177-464d-8445-004488912371" path="/var/lib/kubelet/pods/b129ac68-f177-464d-8445-004488912371/volumes" Sep 14 02:03:01.830787 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.830749 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:03:01.831344 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831323 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25d9cda-51ad-47e6-b126-083a5808b163" containerName="test-curl-7446-7f215c" Sep 14 02:03:01.831344 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831344 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25d9cda-51ad-47e6-b126-083a5808b163" containerName="test-curl-7446-7f215c" Sep 14 02:03:01.831499 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831424 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b129ac68-f177-464d-8445-004488912371" containerName="main" Sep 14 02:03:01.831499 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831453 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b129ac68-f177-464d-8445-004488912371" containerName="main" Sep 14 02:03:01.831615 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831558 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25d9cda-51ad-47e6-b126-083a5808b163" containerName="test-curl-7446-7f215c" Sep 14 02:03:01.831615 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.831574 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b129ac68-f177-464d-8445-004488912371" containerName="main" Sep 14 02:03:01.834350 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.834329 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.836877 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.836855 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-39c298-openshift-default-dockercfg-7ndp2\"" Sep 14 02:03:01.846239 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.846216 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:03:01.886739 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886855 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.886921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.887185 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886934 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.887185 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.886995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988306 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988323 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988382 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988431 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988468 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988506 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988500 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988876 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988876 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988747 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.988876 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988777 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.989019 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988882 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.989019 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.988957 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.989108 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.989043 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.991058 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.991034 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.991174 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.991156 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.995698 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.995672 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:01.995917 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:01.995895 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5\") pod \"e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:02.145482 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:02.145457 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:02.417805 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:02.417774 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:03:02.419834 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:03:02.419809 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd4d287_a1cb_4616_9e11_311188586426.slice/crio-b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20 WatchSource:0}: Error finding container b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20: Status 404 returned error can't find the container with id b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20 Sep 14 02:03:02.422019 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:02.421984 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:02.422099 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:02.422053 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:02.422099 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:02.422082 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:03.283308 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:03.283239 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerStarted","Data":"ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3"} Sep 14 02:03:03.283716 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:03.283326 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerStarted","Data":"b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20"} Sep 14 02:03:03.301476 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:03.301421 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podStartSLOduration=2.301404026 podStartE2EDuration="2.301404026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:03.300240975 +0000 UTC m=+1929.973357161" watchObservedRunningTime="2026-09-14 02:03:03.301404026 +0000 UTC m=+1929.974520200" Sep 14 02:03:04.146548 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:04.146516 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:04.152451 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:04.152420 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:04.288852 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:04.288821 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:04.289893 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:04.289867 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:07.006746 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.006714 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:03:07.010623 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.010601 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.012691 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.012668 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-39c298-serving-cert\"" Sep 14 02:03:07.020667 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.020644 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:03:07.040302 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.040275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.040450 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.040308 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.040450 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.040349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699h8\" (UniqueName: \"kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.040450 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.040382 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.141611 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.141578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.141611 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.141614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.141863 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.141645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699h8\" (UniqueName: \"kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.141863 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.141666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.142483 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.142453 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.144345 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.144319 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.144441 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.144398 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.150170 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.150146 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-699h8\" (UniqueName: \"kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8\") pod \"maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.324233 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.324156 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:07.601216 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.600556 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:03:07.601959 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:03:07.601931 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a22707_7f0b_4684_a7e3_aafba5ae44e4.slice/crio-a95311329db43fc71bd4bbd113eac279c9962a31836b146458f35299c6b48fda WatchSource:0}: Error finding container a95311329db43fc71bd4bbd113eac279c9962a31836b146458f35299c6b48fda: Status 404 returned error can't find the container with id a95311329db43fc71bd4bbd113eac279c9962a31836b146458f35299c6b48fda Sep 14 02:03:07.692062 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.692032 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:03:07.696786 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.696771 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.699906 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.699882 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-39c298-dockercfg-792sp\"" Sep 14 02:03:07.700043 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.699942 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-39c298\"" Sep 14 02:03:07.707419 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.707397 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:03:07.747933 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.747908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk64\" (UniqueName: \"kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.748060 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.747953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.849352 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.849268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk64\" (UniqueName: \"kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.849352 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.849335 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.850037 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.850018 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:07.856982 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:07.856947 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk64\" (UniqueName: \"kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64\") pod \"payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:08.008913 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.008879 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:08.282146 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.282052 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:03:08.285268 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:03:08.285226 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda600035d_e1e8_43de_b4fe_926f66420913.slice/crio-b8999002d676defe3149334e650387996acd2505b9d793147fd2ccaef01f3ef6 WatchSource:0}: Error finding container b8999002d676defe3149334e650387996acd2505b9d793147fd2ccaef01f3ef6: Status 404 returned error can't find the container with id b8999002d676defe3149334e650387996acd2505b9d793147fd2ccaef01f3ef6 Sep 14 02:03:08.310872 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.310828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" event={"ID":"a600035d-e1e8-43de-b4fe-926f66420913","Type":"ContainerStarted","Data":"b8999002d676defe3149334e650387996acd2505b9d793147fd2ccaef01f3ef6"} Sep 14 02:03:08.312846 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.312817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" event={"ID":"c7a22707-7f0b-4684-a7e3-aafba5ae44e4","Type":"ContainerStarted","Data":"1aa8bc064d77ea474d248183fb76812fbe78e623894863e4b4bfea6c83f33fe9"} Sep 14 02:03:08.312978 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.312869 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" event={"ID":"c7a22707-7f0b-4684-a7e3-aafba5ae44e4","Type":"ContainerStarted","Data":"a95311329db43fc71bd4bbd113eac279c9962a31836b146458f35299c6b48fda"} Sep 14 02:03:08.312978 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.312896 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:08.328299 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:08.328240 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" podStartSLOduration=1.864610387 podStartE2EDuration="2.328225473s" totalImagesPullingTime="463.615086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:08.327768175 +0000 UTC m=+1935.000884350" watchObservedRunningTime="2026-09-14 02:03:08.328225473 +0000 UTC m=+1935.001341647" Sep 14 02:03:09.318776 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:09.318734 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" event={"ID":"a600035d-e1e8-43de-b4fe-926f66420913","Type":"ContainerStarted","Data":"c53d2363bfaccb152185fcb361e9c5106ab3ecb574538d09fce0928b00128030"} Sep 14 02:03:09.319170 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:09.318831 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:09.333459 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:09.333412 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" podStartSLOduration=2.3333953640000002 podStartE2EDuration="2.333395364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:03:09.332732674 +0000 UTC m=+1936.005848846" watchObservedRunningTime="2026-09-14 02:03:09.333395364 +0000 UTC m=+1936.006511542" Sep 14 02:03:14.325014 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:14.324982 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:03:20.325706 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:20.325676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:03:20.330379 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:20.330351 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 02:03:20.330505 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:20.330403 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 02:03:21.382661 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.382632 2573 generic.go:356] "Generic (PLEG): container finished" podID="5bd4d287-a1cb-4616-9e11-311188586426" containerID="ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3" exitCode=0 Sep 14 02:03:21.383059 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.382709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerDied","Data":"ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3"} Sep 14 02:03:21.383200 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.383182 2573 scope.go:117] "RemoveContainer" containerID="ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3" Sep 14 02:03:21.384344 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.384311 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:21.384402 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.384389 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:21.384445 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:21.384430 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 02:03:22.389921 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:22.389881 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerStarted","Data":"96bb4efa4b4392cb4bbcc376bbfea36ed533398c7bf71ff5fc4ef9e917fc5db7"} Sep 14 02:03:23.146070 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:23.146038 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:23.146359 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:23.146333 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 02:03:23.146463 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:23.146391 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 02:03:24.146904 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:24.146866 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 02:03:24.147442 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:24.146934 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 02:03:25.146602 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:25.146572 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 02:03:25.146841 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:25.146629 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 02:03:26.146942 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:26.146904 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 02:03:26.147382 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:26.146966 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 02:03:27.146524 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:27.146485 2573 patch_prober.go:28] interesting pod/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 02:03:27.146714 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:27.146546 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 02:03:28.150096 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:28.150064 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:28.150534 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:28.150376 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:03:28.151177 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:03:28.151154 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:04:04.747580 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:04:04.747543 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:05:06.268826 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:06.268788 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp_a600035d-e1e8-43de-b4fe-926f66420913/payload-processing/0.log" Sep 14 02:05:06.401698 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:06.401667 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:05:13.593837 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.593799 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:13.597468 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.597447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.600040 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.600020 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-072cc4-dockercfg-gdstt\"" Sep 14 02:05:13.600149 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.600082 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-072cc4\"" Sep 14 02:05:13.607067 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.607038 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:13.631797 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.631768 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:13.635186 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.635169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.643951 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.643931 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:13.697923 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.697893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.698142 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.697949 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf29\" (UniqueName: \"kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.799439 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.799401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2lfs\" (UniqueName: \"kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.799608 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.799453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf29\" (UniqueName: \"kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.799608 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.799582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.799699 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.799623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.800406 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.800387 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.808778 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.808750 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf29\" (UniqueName: \"kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29\") pod \"payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.900067 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.900037 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.900295 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.900087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2lfs\" (UniqueName: \"kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.900792 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.900764 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.907759 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.907733 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2lfs\" (UniqueName: \"kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs\") pod \"payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:13.909421 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.909395 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:13.947633 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:13.947606 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:14.194425 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.194394 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:14.195651 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:05:14.195618 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae379dd_2685_48ee_b64e_6fc8aa0f3cae.slice/crio-b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9 WatchSource:0}: Error finding container b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9: Status 404 returned error can't find the container with id b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9 Sep 14 02:05:14.237018 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.236986 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:14.243377 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:05:14.243350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d1108e_fbe0_4919_adb6_6e266b5f567a.slice/crio-72c36fc266818d0a1ba17d2b6969d884dd5590b86c348106f2cecd91b0c02a79 WatchSource:0}: Error finding container 72c36fc266818d0a1ba17d2b6969d884dd5590b86c348106f2cecd91b0c02a79: Status 404 returned error can't find the container with id 72c36fc266818d0a1ba17d2b6969d884dd5590b86c348106f2cecd91b0c02a79 Sep 14 02:05:14.945672 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.945631 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" event={"ID":"00d1108e-fbe0-4919-adb6-6e266b5f567a","Type":"ContainerStarted","Data":"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8"} Sep 14 02:05:14.946093 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.945684 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" event={"ID":"00d1108e-fbe0-4919-adb6-6e266b5f567a","Type":"ContainerStarted","Data":"72c36fc266818d0a1ba17d2b6969d884dd5590b86c348106f2cecd91b0c02a79"} Sep 14 02:05:14.946093 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.945778 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:14.947471 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.947447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" event={"ID":"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae","Type":"ContainerStarted","Data":"6ef35529422374f6a4b3429f4bc932bb58efbb4576dca4a55f32faa0f4efb34f"} Sep 14 02:05:14.947582 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.947483 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" event={"ID":"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae","Type":"ContainerStarted","Data":"b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9"} Sep 14 02:05:14.947582 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.947513 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:14.962120 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.962072 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" podStartSLOduration=1.962056387 podStartE2EDuration="1.962056387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:05:14.961818207 +0000 UTC m=+2061.634934382" watchObservedRunningTime="2026-09-14 02:05:14.962056387 +0000 UTC m=+2061.635172560" Sep 14 02:05:14.977076 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:14.977037 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" podStartSLOduration=1.97702615 podStartE2EDuration="1.97702615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:05:14.975882212 +0000 UTC m=+2061.648998388" watchObservedRunningTime="2026-09-14 02:05:14.97702615 +0000 UTC m=+2061.650142325" Sep 14 02:05:25.954739 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:25.954706 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:25.955280 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:25.954876 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:51.834179 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:51.834146 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:51.834759 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:51.834446 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" podUID="00d1108e-fbe0-4919-adb6-6e266b5f567a" containerName="payload-processing" containerID="cri-o://e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8" gracePeriod=30 Sep 14 02:05:51.840084 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:51.840054 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:51.840339 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:51.840296 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" podUID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" containerName="payload-pre-processing" containerID="cri-o://6ef35529422374f6a4b3429f4bc932bb58efbb4576dca4a55f32faa0f4efb34f" gracePeriod=30 Sep 14 02:05:52.134052 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.134019 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:52.137565 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.137538 2573 generic.go:356] "Generic (PLEG): container finished" podID="00d1108e-fbe0-4919-adb6-6e266b5f567a" containerID="e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8" exitCode=0 Sep 14 02:05:52.137684 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.137588 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" Sep 14 02:05:52.137684 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.137604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" event={"ID":"00d1108e-fbe0-4919-adb6-6e266b5f567a","Type":"ContainerDied","Data":"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8"} Sep 14 02:05:52.137684 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.137653 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575" event={"ID":"00d1108e-fbe0-4919-adb6-6e266b5f567a","Type":"ContainerDied","Data":"72c36fc266818d0a1ba17d2b6969d884dd5590b86c348106f2cecd91b0c02a79"} Sep 14 02:05:52.137824 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.137682 2573 scope.go:117] "RemoveContainer" containerID="e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8" Sep 14 02:05:52.138825 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.138806 2573 generic.go:356] "Generic (PLEG): container finished" podID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" containerID="6ef35529422374f6a4b3429f4bc932bb58efbb4576dca4a55f32faa0f4efb34f" exitCode=0 Sep 14 02:05:52.138901 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.138883 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" event={"ID":"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae","Type":"ContainerDied","Data":"6ef35529422374f6a4b3429f4bc932bb58efbb4576dca4a55f32faa0f4efb34f"} Sep 14 02:05:52.138938 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.138925 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" event={"ID":"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae","Type":"ContainerDied","Data":"b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9"} Sep 14 02:05:52.138983 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.138942 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2b9baaa930eebcc0c544faf469eb7c923ab38eb89f93c2b6a908c952777fdc9" Sep 14 02:05:52.139942 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.139926 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:52.149549 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.149337 2573 scope.go:117] "RemoveContainer" containerID="e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8" Sep 14 02:05:52.150157 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:05:52.150122 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8\": container with ID starting with e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8 not found: ID does not exist" containerID="e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8" Sep 14 02:05:52.150741 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.150158 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8"} err="failed to get container status \"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8\": rpc error: code = NotFound desc = could not find container \"e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8\": container with ID starting with e1618191603dcebe1c21b7a925166def5c9d74227e7594a0b835f4ad4f81a6a8 not found: ID does not exist" Sep 14 02:05:52.283455 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283404 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume\") pod \"00d1108e-fbe0-4919-adb6-6e266b5f567a\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " Sep 14 02:05:52.283641 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283484 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume\") pod \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " Sep 14 02:05:52.283641 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283528 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svf29\" (UniqueName: \"kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29\") pod \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\" (UID: \"1ae379dd-2685-48ee-b64e-6fc8aa0f3cae\") " Sep 14 02:05:52.283641 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2lfs\" (UniqueName: \"kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs\") pod \"00d1108e-fbe0-4919-adb6-6e266b5f567a\" (UID: \"00d1108e-fbe0-4919-adb6-6e266b5f567a\") " Sep 14 02:05:52.283838 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283814 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00d1108e-fbe0-4919-adb6-6e266b5f567a" (UID: "00d1108e-fbe0-4919-adb6-6e266b5f567a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:05:52.283917 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283861 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" (UID: "1ae379dd-2685-48ee-b64e-6fc8aa0f3cae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:05:52.283917 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.283901 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00d1108e-fbe0-4919-adb6-6e266b5f567a-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:05:52.285812 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.285789 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29" (OuterVolumeSpecName: "kube-api-access-svf29") pod "1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" (UID: "1ae379dd-2685-48ee-b64e-6fc8aa0f3cae"). InnerVolumeSpecName "kube-api-access-svf29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:05:52.285891 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.285826 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs" (OuterVolumeSpecName: "kube-api-access-n2lfs") pod "00d1108e-fbe0-4919-adb6-6e266b5f567a" (UID: "00d1108e-fbe0-4919-adb6-6e266b5f567a"). InnerVolumeSpecName "kube-api-access-n2lfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:05:52.384817 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.384779 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2lfs\" (UniqueName: \"kubernetes.io/projected/00d1108e-fbe0-4919-adb6-6e266b5f567a-kube-api-access-n2lfs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:05:52.384817 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.384816 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:05:52.384997 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.384829 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svf29\" (UniqueName: \"kubernetes.io/projected/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae-kube-api-access-svf29\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:05:52.458810 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.458776 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:52.462539 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:52.462517 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-072cc4-5ff548fd45-cp575"] Sep 14 02:05:53.144835 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:53.144806 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j" Sep 14 02:05:53.164835 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:53.164806 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:53.168483 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:53.168458 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-072cc4-6c86694d58-7965j"] Sep 14 02:05:53.864225 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:53.864190 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d1108e-fbe0-4919-adb6-6e266b5f567a" path="/var/lib/kubelet/pods/00d1108e-fbe0-4919-adb6-6e266b5f567a/volumes" Sep 14 02:05:53.864702 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:05:53.864679 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" path="/var/lib/kubelet/pods/1ae379dd-2685-48ee-b64e-6fc8aa0f3cae/volumes" Sep 14 02:06:19.040233 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.040175 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:06:19.041095 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.041029 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerName="maas-api" containerID="cri-o://1aa8bc064d77ea474d248183fb76812fbe78e623894863e4b4bfea6c83f33fe9" gracePeriod=30 Sep 14 02:06:19.089003 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.088972 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:06:19.089215 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.089190 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" podUID="a600035d-e1e8-43de-b4fe-926f66420913" containerName="payload-processing" containerID="cri-o://c53d2363bfaccb152185fcb361e9c5106ab3ecb574538d09fce0928b00128030" gracePeriod=30 Sep 14 02:06:19.281912 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.281684 2573 generic.go:356] "Generic (PLEG): container finished" podID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerID="1aa8bc064d77ea474d248183fb76812fbe78e623894863e4b4bfea6c83f33fe9" exitCode=0 Sep 14 02:06:19.281912 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.281828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" event={"ID":"c7a22707-7f0b-4684-a7e3-aafba5ae44e4","Type":"ContainerDied","Data":"1aa8bc064d77ea474d248183fb76812fbe78e623894863e4b4bfea6c83f33fe9"} Sep 14 02:06:19.283991 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.283962 2573 generic.go:356] "Generic (PLEG): container finished" podID="a600035d-e1e8-43de-b4fe-926f66420913" containerID="c53d2363bfaccb152185fcb361e9c5106ab3ecb574538d09fce0928b00128030" exitCode=0 Sep 14 02:06:19.284149 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.284002 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" event={"ID":"a600035d-e1e8-43de-b4fe-926f66420913","Type":"ContainerDied","Data":"c53d2363bfaccb152185fcb361e9c5106ab3ecb574538d09fce0928b00128030"} Sep 14 02:06:19.366629 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.366594 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:06:19.393694 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.393664 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:06:19.435655 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.435629 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca\") pod \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " Sep 14 02:06:19.435805 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.435665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-699h8\" (UniqueName: \"kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8\") pod \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " Sep 14 02:06:19.435805 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.435699 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls\") pod \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " Sep 14 02:06:19.435805 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.435721 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs\") pod \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\" (UID: \"c7a22707-7f0b-4684-a7e3-aafba5ae44e4\") " Sep 14 02:06:19.435991 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.435975 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7a22707-7f0b-4684-a7e3-aafba5ae44e4" (UID: "c7a22707-7f0b-4684-a7e3-aafba5ae44e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:06:19.436178 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.436159 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.438002 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.437978 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8" (OuterVolumeSpecName: "kube-api-access-699h8") pod "c7a22707-7f0b-4684-a7e3-aafba5ae44e4" (UID: "c7a22707-7f0b-4684-a7e3-aafba5ae44e4"). InnerVolumeSpecName "kube-api-access-699h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:06:19.438002 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.437977 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7a22707-7f0b-4684-a7e3-aafba5ae44e4" (UID: "c7a22707-7f0b-4684-a7e3-aafba5ae44e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:06:19.438132 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.437993 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7a22707-7f0b-4684-a7e3-aafba5ae44e4" (UID: "c7a22707-7f0b-4684-a7e3-aafba5ae44e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:06:19.536866 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.536835 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gk64\" (UniqueName: \"kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64\") pod \"a600035d-e1e8-43de-b4fe-926f66420913\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " Sep 14 02:06:19.537055 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.536951 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume\") pod \"a600035d-e1e8-43de-b4fe-926f66420913\" (UID: \"a600035d-e1e8-43de-b4fe-926f66420913\") " Sep 14 02:06:19.537316 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.537296 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-699h8\" (UniqueName: \"kubernetes.io/projected/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-kube-api-access-699h8\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.537397 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.537322 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.537397 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.537336 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a22707-7f0b-4684-a7e3-aafba5ae44e4-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.537397 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.537377 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a600035d-e1e8-43de-b4fe-926f66420913" (UID: "a600035d-e1e8-43de-b4fe-926f66420913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:06:19.539022 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.538998 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64" (OuterVolumeSpecName: "kube-api-access-9gk64") pod "a600035d-e1e8-43de-b4fe-926f66420913" (UID: "a600035d-e1e8-43de-b4fe-926f66420913"). InnerVolumeSpecName "kube-api-access-9gk64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:06:19.638478 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.638440 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk64\" (UniqueName: \"kubernetes.io/projected/a600035d-e1e8-43de-b4fe-926f66420913-kube-api-access-9gk64\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:19.638478 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:19.638473 2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a600035d-e1e8-43de-b4fe-926f66420913-plugins-config-volume\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:06:20.290062 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.290019 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" event={"ID":"a600035d-e1e8-43de-b4fe-926f66420913","Type":"ContainerDied","Data":"b8999002d676defe3149334e650387996acd2505b9d793147fd2ccaef01f3ef6"} Sep 14 02:06:20.290062 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.290048 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp" Sep 14 02:06:20.290616 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.290095 2573 scope.go:117] "RemoveContainer" containerID="c53d2363bfaccb152185fcb361e9c5106ab3ecb574538d09fce0928b00128030" Sep 14 02:06:20.291635 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.291521 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" event={"ID":"c7a22707-7f0b-4684-a7e3-aafba5ae44e4","Type":"ContainerDied","Data":"a95311329db43fc71bd4bbd113eac279c9962a31836b146458f35299c6b48fda"} Sep 14 02:06:20.291635 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.291559 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" Sep 14 02:06:20.299117 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.299102 2573 scope.go:117] "RemoveContainer" containerID="1aa8bc064d77ea474d248183fb76812fbe78e623894863e4b4bfea6c83f33fe9" Sep 14 02:06:20.308536 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.308513 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:06:20.312512 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.312491 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-39c298-b7b7897bf-5x8tp"] Sep 14 02:06:20.322388 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.322366 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:06:20.327887 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:20.327867 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd"] Sep 14 02:06:21.861514 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:21.861478 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a600035d-e1e8-43de-b4fe-926f66420913" path="/var/lib/kubelet/pods/a600035d-e1e8-43de-b4fe-926f66420913/volumes" Sep 14 02:06:21.861901 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:21.861847 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" path="/var/lib/kubelet/pods/c7a22707-7f0b-4684-a7e3-aafba5ae44e4/volumes" Sep 14 02:06:22.320316 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:06:22.320273 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-39c298-67cf7bd4d-gmlnd" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.67:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 14 02:07:07.299900 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:07.299868 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:07:07.300535 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:07.300066 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" containerID="cri-o://96bb4efa4b4392cb4bbcc376bbfea36ed533398c7bf71ff5fc4ef9e917fc5db7" gracePeriod=30 Sep 14 02:07:12.548166 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.548138 2573 generic.go:356] "Generic (PLEG): container finished" podID="5bd4d287-a1cb-4616-9e11-311188586426" containerID="96bb4efa4b4392cb4bbcc376bbfea36ed533398c7bf71ff5fc4ef9e917fc5db7" exitCode=0 Sep 14 02:07:12.548544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.548214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerDied","Data":"96bb4efa4b4392cb4bbcc376bbfea36ed533398c7bf71ff5fc4ef9e917fc5db7"} Sep 14 02:07:12.548544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.548295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" event={"ID":"5bd4d287-a1cb-4616-9e11-311188586426","Type":"ContainerDied","Data":"b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20"} Sep 14 02:07:12.548544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.548317 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b79d0f980193e6101ea64fbb77b8aea7e32922d5c59c4121ba82cd16dc250a20" Sep 14 02:07:12.548544 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.548332 2573 scope.go:117] "RemoveContainer" containerID="ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3" Sep 14 02:07:12.560464 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:12.560444 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3\": container with ID starting with ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3 not found: ID does not exist" containerID="ab48f272bd515299e623e38b715e5310bb57883cfa62129b8b60b526ae3e42d3" Sep 14 02:07:12.563934 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.563894 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:07:12.645740 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645713 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.645740 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645743 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.645940 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645774 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.645940 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645807 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.645940 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645848 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.645940 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645936 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.646148 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.645967 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.646148 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646035 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.646148 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646065 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket\") pod \"5bd4d287-a1cb-4616-9e11-311188586426\" (UID: \"5bd4d287-a1cb-4616-9e11-311188586426\") " Sep 14 02:07:12.646356 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646161 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data" (OuterVolumeSpecName: "istio-data") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:12.646356 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646211 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:12.646537 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646500 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:12.646537 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:12.646646 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646590 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.646646 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646609 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.646646 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.646623 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:12.647993 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.647962 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:07:12.648105 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.648029 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5" (OuterVolumeSpecName: "kube-api-access-2k7r5") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "kube-api-access-2k7r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:12.648524 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.648502 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:12.648622 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.648578 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bd4d287-a1cb-4616-9e11-311188586426" (UID: "5bd4d287-a1cb-4616-9e11-311188586426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:12.747078 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747043 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k7r5\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-kube-api-access-2k7r5\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747078 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747073 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd4d287-a1cb-4616-9e11-311188586426-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747286 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747086 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd4d287-a1cb-4616-9e11-311188586426-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747286 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747098 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747286 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747110 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747286 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747121 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd4d287-a1cb-4616-9e11-311188586426-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:12.747286 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:12.747131 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd4d287-a1cb-4616-9e11-311188586426-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:13.554508 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:13.554479 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x" Sep 14 02:07:13.575022 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:13.574995 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:07:13.578183 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:13.578162 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-39c298-openshift-default-6b65d754f8-6m58x"] Sep 14 02:07:13.861395 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:13.861304 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd4d287-a1cb-4616-9e11-311188586426" path="/var/lib/kubelet/pods/5bd4d287-a1cb-4616-9e11-311188586426/volumes" Sep 14 02:07:41.798801 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:41.798759 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 02:07:41.799244 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:41.799012 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" containerID="cri-o://9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61" gracePeriod=30 Sep 14 02:07:47.061545 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.061518 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 02:07:47.169126 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169096 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169126 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169127 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169172 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169217 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169274 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tb2w\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169302 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169325 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169399 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169362 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169711 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169422 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data\") pod \"178ecc4b-0e2c-462e-bc77-f86728427097\" (UID: \"178ecc4b-0e2c-462e-bc77-f86728427097\") " Sep 14 02:07:47.169711 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169533 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:47.169711 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169689 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:47.169863 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169728 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data" (OuterVolumeSpecName: "istio-data") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:47.169863 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169847 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:47.170011 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.169980 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:47.170129 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.170072 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-data\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.170129 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.170099 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/178ecc4b-0e2c-462e-bc77-f86728427097-istiod-ca-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.170129 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.170117 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.170299 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.170129 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-credential-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.171809 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.171781 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:07:47.171910 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.171872 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 02:07:47.172197 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.172170 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token" (OuterVolumeSpecName: "istio-token") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:47.172302 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.172170 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w" (OuterVolumeSpecName: "kube-api-access-4tb2w") pod "178ecc4b-0e2c-462e-bc77-f86728427097" (UID: "178ecc4b-0e2c-462e-bc77-f86728427097"). InnerVolumeSpecName "kube-api-access-4tb2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:47.271461 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.271431 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-istio-token\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.271461 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.271457 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/178ecc4b-0e2c-462e-bc77-f86728427097-istio-podinfo\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.271461 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.271466 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tb2w\" (UniqueName: \"kubernetes.io/projected/178ecc4b-0e2c-462e-bc77-f86728427097-kube-api-access-4tb2w\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.271685 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.271475 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-workload-socket\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.271685 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.271485 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/178ecc4b-0e2c-462e-bc77-f86728427097-istio-envoy\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:47.725743 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.725709 2573 generic.go:356] "Generic (PLEG): container finished" podID="178ecc4b-0e2c-462e-bc77-f86728427097" containerID="9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61" exitCode=0 Sep 14 02:07:47.725907 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.725760 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" Sep 14 02:07:47.725907 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.725794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerDied","Data":"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61"} Sep 14 02:07:47.725907 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.725857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2" event={"ID":"178ecc4b-0e2c-462e-bc77-f86728427097","Type":"ContainerDied","Data":"a539ed54f41cabfebe2589e750949cd810d122963200330ad19662000b6ca1f1"} Sep 14 02:07:47.725907 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.725883 2573 scope.go:117] "RemoveContainer" containerID="9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61" Sep 14 02:07:47.735566 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.735548 2573 scope.go:117] "RemoveContainer" containerID="0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac" Sep 14 02:07:47.743879 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.743859 2573 scope.go:117] "RemoveContainer" containerID="9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61" Sep 14 02:07:47.744241 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:47.744211 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61\": container with ID starting with 9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61 not found: ID does not exist" containerID="9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61" Sep 14 02:07:47.744374 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.744270 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61"} err="failed to get container status \"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61\": rpc error: code = NotFound desc = could not find container \"9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61\": container with ID starting with 9ec5a7800874bb4ea6136de674d94293958600b84b4a8e52841aeca9eefccd61 not found: ID does not exist" Sep 14 02:07:47.744374 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.744293 2573 scope.go:117] "RemoveContainer" containerID="0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac" Sep 14 02:07:47.744787 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:47.744761 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac\": container with ID starting with 0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac not found: ID does not exist" containerID="0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac" Sep 14 02:07:47.744912 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.744793 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac"} err="failed to get container status \"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac\": rpc error: code = NotFound desc = could not find container \"0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac\": container with ID starting with 0d258e5a736c57d927ffdf654972ff1dd173e06625b3cf6f80f1ed380f1fd5ac not found: ID does not exist" Sep 14 02:07:47.746824 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.746805 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 02:07:47.750541 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.750521 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-5d3ac7-openshift-default-69548df498-xm9h2"] Sep 14 02:07:47.862689 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:47.862651 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" path="/var/lib/kubelet/pods/178ecc4b-0e2c-462e-bc77-f86728427097/volumes" Sep 14 02:07:52.766307 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:52.766264 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 02:07:52.766796 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:52.766544 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" podUID="1f212633-1f9c-46d2-be62-ac16807ac837" containerName="maas-api" containerID="cri-o://76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96" gracePeriod=30 Sep 14 02:07:53.018230 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.018163 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 02:07:53.026918 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.026892 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls\") pod \"1f212633-1f9c-46d2-be62-ac16807ac837\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " Sep 14 02:07:53.027067 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.026928 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca\") pod \"1f212633-1f9c-46d2-be62-ac16807ac837\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " Sep 14 02:07:53.027067 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.026968 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9mg7\" (UniqueName: \"kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7\") pod \"1f212633-1f9c-46d2-be62-ac16807ac837\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " Sep 14 02:07:53.027067 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.027021 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs\") pod \"1f212633-1f9c-46d2-be62-ac16807ac837\" (UID: \"1f212633-1f9c-46d2-be62-ac16807ac837\") " Sep 14 02:07:53.027377 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.027348 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1f212633-1f9c-46d2-be62-ac16807ac837" (UID: "1f212633-1f9c-46d2-be62-ac16807ac837"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:07:53.029359 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.029336 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1f212633-1f9c-46d2-be62-ac16807ac837" (UID: "1f212633-1f9c-46d2-be62-ac16807ac837"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:07:53.029803 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.029782 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1f212633-1f9c-46d2-be62-ac16807ac837" (UID: "1f212633-1f9c-46d2-be62-ac16807ac837"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:07:53.029877 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.029786 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7" (OuterVolumeSpecName: "kube-api-access-r9mg7") pod "1f212633-1f9c-46d2-be62-ac16807ac837" (UID: "1f212633-1f9c-46d2-be62-ac16807ac837"). InnerVolumeSpecName "kube-api-access-r9mg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:07:53.128268 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.128220 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.128268 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.128272 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f212633-1f9c-46d2-be62-ac16807ac837-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.128459 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.128284 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f212633-1f9c-46d2-be62-ac16807ac837-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.128459 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.128296 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9mg7\" (UniqueName: \"kubernetes.io/projected/1f212633-1f9c-46d2-be62-ac16807ac837-kube-api-access-r9mg7\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:07:53.757493 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.757461 2573 generic.go:356] "Generic (PLEG): container finished" podID="1f212633-1f9c-46d2-be62-ac16807ac837" containerID="76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96" exitCode=0 Sep 14 02:07:53.757687 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.757503 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" Sep 14 02:07:53.757687 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.757555 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" event={"ID":"1f212633-1f9c-46d2-be62-ac16807ac837","Type":"ContainerDied","Data":"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96"} Sep 14 02:07:53.757687 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.757619 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5" event={"ID":"1f212633-1f9c-46d2-be62-ac16807ac837","Type":"ContainerDied","Data":"8f0b00727187d43ed360af460693ba008c2c92532a44d7cfa6fffbde2e882eec"} Sep 14 02:07:53.757687 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.757651 2573 scope.go:117] "RemoveContainer" containerID="76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96" Sep 14 02:07:53.768446 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.768269 2573 scope.go:117] "RemoveContainer" containerID="76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96" Sep 14 02:07:53.768687 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:53.768554 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96\": container with ID starting with 76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96 not found: ID does not exist" containerID="76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96" Sep 14 02:07:53.768687 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.768580 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96"} err="failed to get container status \"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96\": rpc error: code = NotFound desc = could not find container \"76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96\": container with ID starting with 76984d5eb824b7b1dd43a3ded9e4ae59fbe623970a1eda8c8cc2310dfaf88e96 not found: ID does not exist" Sep 14 02:07:53.781758 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.781733 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 02:07:53.786642 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.786621 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-86995c-768658885d-cz8k5"] Sep 14 02:07:53.867044 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:07:53.867014 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f212633-1f9c-46d2-be62-ac16807ac837" path="/var/lib/kubelet/pods/1f212633-1f9c-46d2-be62-ac16807ac837/volumes" Sep 14 02:07:58.437971 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:58.437939 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:07:58.438353 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:58.437990 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:07:59.612146 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:07:59.612112 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:02.565318 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.565275 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 02:08:02.565782 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.565505 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" podUID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" containerName="manager" containerID="cri-o://aec44580aced7dbaedbdea2d5b5016918f2c695abc7ac7ff0e1aee08e3d9a8f6" gracePeriod=30 Sep 14 02:08:02.801796 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.801765 2573 generic.go:356] "Generic (PLEG): container finished" podID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" containerID="aec44580aced7dbaedbdea2d5b5016918f2c695abc7ac7ff0e1aee08e3d9a8f6" exitCode=0 Sep 14 02:08:02.801924 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.801831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" event={"ID":"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e","Type":"ContainerDied","Data":"aec44580aced7dbaedbdea2d5b5016918f2c695abc7ac7ff0e1aee08e3d9a8f6"} Sep 14 02:08:02.817954 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.817898 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 02:08:02.920641 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.920612 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert\") pod \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " Sep 14 02:08:02.920823 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.920655 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh4bq\" (UniqueName: \"kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq\") pod \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " Sep 14 02:08:02.920823 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.920683 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") pod \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\" (UID: \"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e\") " Sep 14 02:08:02.922887 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.922855 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" (UID: "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:08:02.922998 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.922901 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" (UID: "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:08:02.922998 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:02.922903 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq" (OuterVolumeSpecName: "kube-api-access-sh4bq") pod "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" (UID: "f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e"). InnerVolumeSpecName "kube-api-access-sh4bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:08:03.021702 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.021649 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-webhook-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:08:03.021702 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.021692 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4bq\" (UniqueName: \"kubernetes.io/projected/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-kube-api-access-sh4bq\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:08:03.021702 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.021703 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:08:03.808404 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.808376 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" Sep 14 02:08:03.808902 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.808424 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-8sfc2" event={"ID":"f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e","Type":"ContainerDied","Data":"6acd40d3c0cc5d8caaeb9c7e715d941e1456f40413fc05c74275c7c63c7092ff"} Sep 14 02:08:03.808902 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.808483 2573 scope.go:117] "RemoveContainer" containerID="aec44580aced7dbaedbdea2d5b5016918f2c695abc7ac7ff0e1aee08e3d9a8f6" Sep 14 02:08:03.830141 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.830110 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 02:08:03.833380 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.833353 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-8sfc2"] Sep 14 02:08:03.861107 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:08:03.861071 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" path="/var/lib/kubelet/pods/f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e/volumes" Sep 14 02:08:07.013858 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:07.013824 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:09.620948 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:09.620908 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:19.661913 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:19.661878 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:21.985142 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:21.985104 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:29.702242 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:29.702206 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:37.029140 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:37.029103 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:39.711448 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:39.711407 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:49.752378 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:49.752347 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:08:52.004128 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:08:52.004086 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f212633_1f9c_46d2_be62_ac16807ac837.slice\": RecentStats: unable to find data in memory cache]" Sep 14 02:09:04.620354 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620317 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620661 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" containerName="payload-pre-processing" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620672 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" containerName="payload-pre-processing" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620682 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerName="maas-api" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620688 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerName="maas-api" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620702 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a600035d-e1e8-43de-b4fe-926f66420913" containerName="payload-processing" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620709 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a600035d-e1e8-43de-b4fe-926f66420913" containerName="payload-processing" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620717 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620722 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620730 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d1108e-fbe0-4919-adb6-6e266b5f567a" containerName="payload-processing" Sep 14 02:09:04.620729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620735 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d1108e-fbe0-4919-adb6-6e266b5f567a" containerName="payload-processing" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620745 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" containerName="manager" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620751 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" containerName="manager" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620756 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620762 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620769 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620775 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620783 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f212633-1f9c-46d2-be62-ac16807ac837" containerName="maas-api" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620789 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f212633-1f9c-46d2-be62-ac16807ac837" containerName="maas-api" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620845 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620852 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620856 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f212633-1f9c-46d2-be62-ac16807ac837" containerName="maas-api" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620862 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a600035d-e1e8-43de-b4fe-926f66420913" containerName="payload-processing" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620868 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd4d287-a1cb-4616-9e11-311188586426" containerName="istio-proxy" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620874 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1fd70fd-0a7e-44ae-80f1-68c5dd1f1e3e" containerName="manager" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620878 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a22707-7f0b-4684-a7e3-aafba5ae44e4" containerName="maas-api" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620883 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ae379dd-2685-48ee-b64e-6fc8aa0f3cae" containerName="payload-pre-processing" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620889 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d1108e-fbe0-4919-adb6-6e266b5f567a" containerName="payload-processing" Sep 14 02:09:04.621046 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.620894 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:04.623964 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.623938 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.627466 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.627436 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 02:09:04.627603 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.627436 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 02:09:04.627603 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.627449 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hchhd\"" Sep 14 02:09:04.627603 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.627449 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 02:09:04.633842 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.633819 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:04.769728 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.769688 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.769918 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.769762 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjrk2\" (UniqueName: \"kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.769918 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.769888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.871319 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.871228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.871319 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.871320 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjrk2\" (UniqueName: \"kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.871556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.871406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.873911 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.873880 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.874027 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.873917 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.879069 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.879052 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjrk2\" (UniqueName: \"kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2\") pod \"maas-controller-59dfd7d9fd-82dhw\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:04.934689 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:04.934657 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:05.203116 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:05.203091 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:05.204726 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:09:05.204702 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a12de2_0e1f_4faf_a51e_5003aae4a257.slice/crio-1fd09734ed1005ae0954a7e6ad03075bddcb6305b6ab1dbb751a224eaab131fb WatchSource:0}: Error finding container 1fd09734ed1005ae0954a7e6ad03075bddcb6305b6ab1dbb751a224eaab131fb: Status 404 returned error can't find the container with id 1fd09734ed1005ae0954a7e6ad03075bddcb6305b6ab1dbb751a224eaab131fb Sep 14 02:09:05.206163 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:05.206143 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 02:09:06.108226 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:06.108136 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" event={"ID":"35a12de2-0e1f-4faf-a51e-5003aae4a257","Type":"ContainerStarted","Data":"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead"} Sep 14 02:09:06.108226 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:06.108193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" event={"ID":"35a12de2-0e1f-4faf-a51e-5003aae4a257","Type":"ContainerStarted","Data":"1fd09734ed1005ae0954a7e6ad03075bddcb6305b6ab1dbb751a224eaab131fb"} Sep 14 02:09:06.108639 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:06.108283 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:06.126401 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:06.126354 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" podStartSLOduration=1.672245322 podStartE2EDuration="2.126337636s" totalImagesPullingTime="454.092314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:09:06.124167688 +0000 UTC m=+2292.797283864" watchObservedRunningTime="2026-09-14 02:09:06.126337636 +0000 UTC m=+2292.799453830" Sep 14 02:09:17.115494 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:17.115461 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:26.529986 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.529923 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:26.530583 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.530167 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" podUID="35a12de2-0e1f-4faf-a51e-5003aae4a257" containerName="manager" containerID="cri-o://f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead" gracePeriod=30 Sep 14 02:09:26.777924 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.777899 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:26.866729 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.866648 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjrk2\" (UniqueName: \"kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2\") pod \"35a12de2-0e1f-4faf-a51e-5003aae4a257\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " Sep 14 02:09:26.866866 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.866737 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs\") pod \"35a12de2-0e1f-4faf-a51e-5003aae4a257\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " Sep 14 02:09:26.866866 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.866785 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert\") pod \"35a12de2-0e1f-4faf-a51e-5003aae4a257\" (UID: \"35a12de2-0e1f-4faf-a51e-5003aae4a257\") " Sep 14 02:09:26.868853 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.868821 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35a12de2-0e1f-4faf-a51e-5003aae4a257" (UID: "35a12de2-0e1f-4faf-a51e-5003aae4a257"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:09:26.868977 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.868849 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "35a12de2-0e1f-4faf-a51e-5003aae4a257" (UID: "35a12de2-0e1f-4faf-a51e-5003aae4a257"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:09:26.868977 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.868942 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2" (OuterVolumeSpecName: "kube-api-access-xjrk2") pod "35a12de2-0e1f-4faf-a51e-5003aae4a257" (UID: "35a12de2-0e1f-4faf-a51e-5003aae4a257"). InnerVolumeSpecName "kube-api-access-xjrk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:09:26.967851 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.967813 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjrk2\" (UniqueName: \"kubernetes.io/projected/35a12de2-0e1f-4faf-a51e-5003aae4a257-kube-api-access-xjrk2\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:09:26.967851 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.967851 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:09:26.968044 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:26.967865 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35a12de2-0e1f-4faf-a51e-5003aae4a257-webhook-cert\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:09:27.207383 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.207347 2573 generic.go:356] "Generic (PLEG): container finished" podID="35a12de2-0e1f-4faf-a51e-5003aae4a257" containerID="f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead" exitCode=0 Sep 14 02:09:27.207556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.207411 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" Sep 14 02:09:27.207556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.207435 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" event={"ID":"35a12de2-0e1f-4faf-a51e-5003aae4a257","Type":"ContainerDied","Data":"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead"} Sep 14 02:09:27.207556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.207493 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-82dhw" event={"ID":"35a12de2-0e1f-4faf-a51e-5003aae4a257","Type":"ContainerDied","Data":"1fd09734ed1005ae0954a7e6ad03075bddcb6305b6ab1dbb751a224eaab131fb"} Sep 14 02:09:27.207556 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.207515 2573 scope.go:117] "RemoveContainer" containerID="f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead" Sep 14 02:09:27.216917 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.216900 2573 scope.go:117] "RemoveContainer" containerID="f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead" Sep 14 02:09:27.217196 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:09:27.217176 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead\": container with ID starting with f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead not found: ID does not exist" containerID="f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead" Sep 14 02:09:27.217294 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.217206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead"} err="failed to get container status \"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead\": rpc error: code = NotFound desc = could not find container \"f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead\": container with ID starting with f1b4d7ae76208e41eaf0a549ec6a58f782f364c650b28cfbfb5a614e353d4ead not found: ID does not exist" Sep 14 02:09:27.229137 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.229107 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:27.232660 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.232637 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-82dhw"] Sep 14 02:09:27.860778 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:27.860730 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a12de2-0e1f-4faf-a51e-5003aae4a257" path="/var/lib/kubelet/pods/35a12de2-0e1f-4faf-a51e-5003aae4a257/volumes" Sep 14 02:09:28.267906 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.267871 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-lwkrw"] Sep 14 02:09:28.268232 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.268220 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a12de2-0e1f-4faf-a51e-5003aae4a257" containerName="manager" Sep 14 02:09:28.268232 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.268234 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a12de2-0e1f-4faf-a51e-5003aae4a257" containerName="manager" Sep 14 02:09:28.268377 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.268277 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:28.268377 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.268286 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="178ecc4b-0e2c-462e-bc77-f86728427097" containerName="istio-proxy" Sep 14 02:09:28.268377 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.268374 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a12de2-0e1f-4faf-a51e-5003aae4a257" containerName="manager" Sep 14 02:09:28.271422 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.271402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.274822 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.274796 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 02:09:28.274950 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.274828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 02:09:28.274950 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.274806 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hchhd\"" Sep 14 02:09:28.274950 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.274848 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 02:09:28.278534 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.278103 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlbm\" (UniqueName: \"kubernetes.io/projected/89c66e52-cbfd-46ec-8f21-6f7837854bc0-kube-api-access-jdlbm\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.278534 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.278230 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-webhook-cert\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.278534 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.278302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-metrics-certs\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.280602 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.280581 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-lwkrw"] Sep 14 02:09:28.379237 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.379186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlbm\" (UniqueName: \"kubernetes.io/projected/89c66e52-cbfd-46ec-8f21-6f7837854bc0-kube-api-access-jdlbm\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.379447 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.379300 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-webhook-cert\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.379447 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.379328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-metrics-certs\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.381944 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.381919 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-webhook-cert\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.382073 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.381972 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89c66e52-cbfd-46ec-8f21-6f7837854bc0-metrics-certs\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.386246 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.386223 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlbm\" (UniqueName: \"kubernetes.io/projected/89c66e52-cbfd-46ec-8f21-6f7837854bc0-kube-api-access-jdlbm\") pod \"maas-controller-59dfd7d9fd-lwkrw\" (UID: \"89c66e52-cbfd-46ec-8f21-6f7837854bc0\") " pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.583035 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.582937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:28.852466 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:28.852441 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59dfd7d9fd-lwkrw"] Sep 14 02:09:28.854734 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:09:28.854707 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c66e52_cbfd_46ec_8f21_6f7837854bc0.slice/crio-69d58c29e9faccdc40a2acd9319f16f23455d456eeb14548a178640cf2a40196 WatchSource:0}: Error finding container 69d58c29e9faccdc40a2acd9319f16f23455d456eeb14548a178640cf2a40196: Status 404 returned error can't find the container with id 69d58c29e9faccdc40a2acd9319f16f23455d456eeb14548a178640cf2a40196 Sep 14 02:09:29.219283 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:29.219243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" event={"ID":"89c66e52-cbfd-46ec-8f21-6f7837854bc0","Type":"ContainerStarted","Data":"69d58c29e9faccdc40a2acd9319f16f23455d456eeb14548a178640cf2a40196"} Sep 14 02:09:30.225581 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:30.225545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" event={"ID":"89c66e52-cbfd-46ec-8f21-6f7837854bc0","Type":"ContainerStarted","Data":"d67ce934eb66897115b09ab6583c2fe7c956aa15b678503e80dcda2812987370"} Sep 14 02:09:30.225996 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:30.225625 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:30.241923 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:30.241878 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" podStartSLOduration=1.904675755 podStartE2EDuration="2.241860027s" totalImagesPullingTime="337.184272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:09:30.240129498 +0000 UTC m=+2316.913245673" watchObservedRunningTime="2026-09-14 02:09:30.241860027 +0000 UTC m=+2316.914976202" Sep 14 02:09:41.232129 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:41.232099 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59dfd7d9fd-lwkrw" Sep 14 02:09:54.176694 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:09:54.176617 2573 scope.go:117] "RemoveContainer" containerID="96bb4efa4b4392cb4bbcc376bbfea36ed533398c7bf71ff5fc4ef9e917fc5db7" Sep 14 02:11:54.206751 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:11:54.206719 2573 scope.go:117] "RemoveContainer" containerID="6ef35529422374f6a4b3429f4bc932bb58efbb4576dca4a55f32faa0f4efb34f" Sep 14 02:13:09.138984 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.138948 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2"] Sep 14 02:13:09.142426 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.142402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.151940 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.151886 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2"] Sep 14 02:13:09.172798 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.172766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj295\" (UniqueName: \"kubernetes.io/projected/e016779c-e434-4b5b-a8dc-4888aa045e64-kube-api-access-cj295\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.172951 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.172848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e016779c-e434-4b5b-a8dc-4888aa045e64-openshift-service-ca\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.172951 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.172871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-metrics-certs\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.172951 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.172928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-maas-api-tls\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.274169 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.274134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e016779c-e434-4b5b-a8dc-4888aa045e64-openshift-service-ca\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.274169 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.274169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-metrics-certs\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.274415 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.274218 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-maas-api-tls\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.274415 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.274291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj295\" (UniqueName: \"kubernetes.io/projected/e016779c-e434-4b5b-a8dc-4888aa045e64-kube-api-access-cj295\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.274860 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.274837 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e016779c-e434-4b5b-a8dc-4888aa045e64-openshift-service-ca\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.277039 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.277013 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-maas-api-tls\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.277122 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.277085 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e016779c-e434-4b5b-a8dc-4888aa045e64-metrics-certs\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.284377 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.284357 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj295\" (UniqueName: \"kubernetes.io/projected/e016779c-e434-4b5b-a8dc-4888aa045e64-kube-api-access-cj295\") pod \"maas-api-794f95dd98-qc6q2\" (UID: \"e016779c-e434-4b5b-a8dc-4888aa045e64\") " pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.456109 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.456032 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:09.725984 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:09.725958 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2"] Sep 14 02:13:09.728316 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:13:09.728292 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode016779c_e434_4b5b_a8dc_4888aa045e64.slice/crio-7872e3dfee50d054cc77516069e4cbaabaed5455f489c8a0387db40a2d48ed19 WatchSource:0}: Error finding container 7872e3dfee50d054cc77516069e4cbaabaed5455f489c8a0387db40a2d48ed19: Status 404 returned error can't find the container with id 7872e3dfee50d054cc77516069e4cbaabaed5455f489c8a0387db40a2d48ed19 Sep 14 02:13:10.241715 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:10.241677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" event={"ID":"e016779c-e434-4b5b-a8dc-4888aa045e64","Type":"ContainerStarted","Data":"86c28af1280392bba38a4188d5bcfec0dbd47ef2b09f8692d57b51e69c4d2745"} Sep 14 02:13:10.242099 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:10.241730 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" event={"ID":"e016779c-e434-4b5b-a8dc-4888aa045e64","Type":"ContainerStarted","Data":"7872e3dfee50d054cc77516069e4cbaabaed5455f489c8a0387db40a2d48ed19"} Sep 14 02:13:10.242099 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:10.241815 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:10.259276 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:10.259203 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" podStartSLOduration=0.884954357 podStartE2EDuration="1.259188176s" totalImagesPullingTime="374.233819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:13:10.258007059 +0000 UTC m=+2536.931123235" watchObservedRunningTime="2026-09-14 02:13:10.259188176 +0000 UTC m=+2536.932304352" Sep 14 02:13:16.253496 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:16.253468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-794f95dd98-qc6q2" Sep 14 02:13:24.029017 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:24.028970 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 02:13:24.031619 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:24.029192 2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" podUID="0342fde0-a4ac-4375-aca8-e3122d747a30" containerName="maas-api" containerID="cri-o://1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a" gracePeriod=30 Sep 14 02:13:25.275155 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.275128 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 02:13:25.313225 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.313190 2573 generic.go:356] "Generic (PLEG): container finished" podID="0342fde0-a4ac-4375-aca8-e3122d747a30" containerID="1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a" exitCode=0 Sep 14 02:13:25.313428 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.313234 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" event={"ID":"0342fde0-a4ac-4375-aca8-e3122d747a30","Type":"ContainerDied","Data":"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a"} Sep 14 02:13:25.313428 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.313239 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" Sep 14 02:13:25.313428 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.313294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c69887854-f5knc" event={"ID":"0342fde0-a4ac-4375-aca8-e3122d747a30","Type":"ContainerDied","Data":"f41aa3646b065916f8888bd5324b9fd985e4b860c02c3200b87f7d47fcbefc46"} Sep 14 02:13:25.313428 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.313319 2573 scope.go:117] "RemoveContainer" containerID="1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a" Sep 14 02:13:25.320014 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.319993 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls\") pod \"0342fde0-a4ac-4375-aca8-e3122d747a30\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " Sep 14 02:13:25.320150 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.320041 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plc8b\" (UniqueName: \"kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b\") pod \"0342fde0-a4ac-4375-aca8-e3122d747a30\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " Sep 14 02:13:25.320150 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.320097 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") pod \"0342fde0-a4ac-4375-aca8-e3122d747a30\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " Sep 14 02:13:25.320150 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.320130 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca\") pod \"0342fde0-a4ac-4375-aca8-e3122d747a30\" (UID: \"0342fde0-a4ac-4375-aca8-e3122d747a30\") " Sep 14 02:13:25.320766 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.320702 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0342fde0-a4ac-4375-aca8-e3122d747a30" (UID: "0342fde0-a4ac-4375-aca8-e3122d747a30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:13:25.322301 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.322242 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0342fde0-a4ac-4375-aca8-e3122d747a30" (UID: "0342fde0-a4ac-4375-aca8-e3122d747a30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:13:25.322850 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.322824 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0342fde0-a4ac-4375-aca8-e3122d747a30" (UID: "0342fde0-a4ac-4375-aca8-e3122d747a30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:13:25.322941 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.322848 2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b" (OuterVolumeSpecName: "kube-api-access-plc8b") pod "0342fde0-a4ac-4375-aca8-e3122d747a30" (UID: "0342fde0-a4ac-4375-aca8-e3122d747a30"). InnerVolumeSpecName "kube-api-access-plc8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:13:25.323184 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.323160 2573 scope.go:117] "RemoveContainer" containerID="1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a" Sep 14 02:13:25.323522 ip-10-0-132-205 kubenswrapper[2573]: E0914 02:13:25.323491 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a\": container with ID starting with 1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a not found: ID does not exist" containerID="1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a" Sep 14 02:13:25.323608 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.323521 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a"} err="failed to get container status \"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a\": rpc error: code = NotFound desc = could not find container \"1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a\": container with ID starting with 1f68f27555ff63173120e46e8ae6fb1703fe914a4c5576fc6d572a40807c7c0a not found: ID does not exist" Sep 14 02:13:25.421543 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.421508 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0342fde0-a4ac-4375-aca8-e3122d747a30-openshift-service-ca\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:13:25.421543 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.421542 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-maas-api-tls\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:13:25.421754 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.421552 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plc8b\" (UniqueName: \"kubernetes.io/projected/0342fde0-a4ac-4375-aca8-e3122d747a30-kube-api-access-plc8b\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:13:25.421754 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.421562 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0342fde0-a4ac-4375-aca8-e3122d747a30-metrics-certs\") on node \"ip-10-0-132-205.ec2.internal\" DevicePath \"\"" Sep 14 02:13:25.635412 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.635378 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 02:13:25.638440 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.638418 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c69887854-f5knc"] Sep 14 02:13:25.861915 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:25.861836 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0342fde0-a4ac-4375-aca8-e3122d747a30" path="/var/lib/kubelet/pods/0342fde0-a4ac-4375-aca8-e3122d747a30/volumes" Sep 14 02:13:43.976756 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:43.976726 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lzzmz_aafa3390-8519-4cb7-8702-32dfdafbf073/manager/0.log" Sep 14 02:13:44.094843 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:44.094793 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gwc5w_f511c637-3e52-4bf2-9d4b-ebace521fecf/manager/0.log" Sep 14 02:13:44.204846 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:44.204814 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59dfd7d9fd-lwkrw_89c66e52-cbfd-46ec-8f21-6f7837854bc0/manager/0.log" Sep 14 02:13:44.529531 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:44.529501 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-7xsrb_7b780b5e-a9f4-4672-8c15-e86b90c37ce6/copy-manifests/0.log" Sep 14 02:13:44.541019 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:44.540997 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-7xsrb_7b780b5e-a9f4-4672-8c15-e86b90c37ce6/manager/0.log" Sep 14 02:13:45.202310 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:45.202281 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-794f95dd98-qc6q2_e016779c-e434-4b5b-a8dc-4888aa045e64/maas-api/0.log" Sep 14 02:13:45.515767 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:45.515675 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zs8vc_e054f313-1f8d-44b8-a5b9-0d592a882809/postgres/0.log" Sep 14 02:13:46.451460 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:46.451430 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pff8c_3f340ab1-9451-42ec-8bc5-fd9834f68b5d/manager/0.log" Sep 14 02:13:46.913927 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:46.913888 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-4m8cz_3e6a1354-3d68-4975-b432-a3e1e84b0330/manager/0.log" Sep 14 02:13:47.668473 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:47.668448 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/1.log" Sep 14 02:13:47.902168 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:47.902140 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:13:48.561527 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:48.561494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-p92wf_e8088731-9d9e-4f70-9a4e-8f2bf63dfa89/main/0.log" Sep 14 02:13:48.764560 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:48.764533 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fmt9tg_0b0646d9-48ef-46ac-afec-86329c61c5a9/main/0.log" Sep 14 02:13:57.553562 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:57.553529 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-25zt6_04aebdb6-17d1-4e99-a6cd-cf767288115f/global-pull-secret-syncer/0.log" Sep 14 02:13:57.714370 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:57.714329 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mq7ns_018a746f-5f8c-474e-94c4-894ba3d0e01d/konnectivity-agent/0.log" Sep 14 02:13:57.807942 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:13:57.807849 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-205.ec2.internal_147f4b079fff27cd4d986e86591ae20a/haproxy/0.log" Sep 14 02:14:02.651414 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:02.651384 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-pff8c_3f340ab1-9451-42ec-8bc5-fd9834f68b5d/manager/0.log" Sep 14 02:14:02.799234 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:02.799189 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-4m8cz_3e6a1354-3d68-4975-b432-a3e1e84b0330/manager/0.log" Sep 14 02:14:04.664911 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:04.664834 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dd679c59-2zsph_eda5e9cb-3fc4-4ae0-a77c-f8882119733f/metrics-server/0.log" Sep 14 02:14:04.813554 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:04.813528 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t6659_8ce5fdda-b5f1-4193-9f6d-3ec443f4378d/node-exporter/0.log" Sep 14 02:14:04.833765 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:04.833742 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t6659_8ce5fdda-b5f1-4193-9f6d-3ec443f4378d/kube-rbac-proxy/0.log" Sep 14 02:14:04.854407 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:04.854387 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t6659_8ce5fdda-b5f1-4193-9f6d-3ec443f4378d/init-textfile/0.log" Sep 14 02:14:05.063187 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.063155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/prometheus/0.log" Sep 14 02:14:05.085385 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.085359 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/config-reloader/0.log" Sep 14 02:14:05.107005 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.106979 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/thanos-sidecar/0.log" Sep 14 02:14:05.131747 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.131721 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/kube-rbac-proxy-web/0.log" Sep 14 02:14:05.154832 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.154814 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/kube-rbac-proxy/0.log" Sep 14 02:14:05.176950 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.176898 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/kube-rbac-proxy-thanos/0.log" Sep 14 02:14:05.201119 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.201094 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1148453c-91a7-4f34-b89b-736843cff462/init-config-reloader/0.log" Sep 14 02:14:05.305635 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.305609 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6759447f-5mwn9_a840611c-42ba-4c5a-b35e-529aa9c28682/telemeter-client/0.log" Sep 14 02:14:05.328217 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.328195 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6759447f-5mwn9_a840611c-42ba-4c5a-b35e-529aa9c28682/reload/0.log" Sep 14 02:14:05.351183 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:05.351157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6759447f-5mwn9_a840611c-42ba-4c5a-b35e-529aa9c28682/kube-rbac-proxy/0.log" Sep 14 02:14:07.881587 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.881554 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj"] Sep 14 02:14:07.881955 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.881932 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0342fde0-a4ac-4375-aca8-e3122d747a30" containerName="maas-api" Sep 14 02:14:07.881955 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.881944 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0342fde0-a4ac-4375-aca8-e3122d747a30" containerName="maas-api" Sep 14 02:14:07.882030 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.882006 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0342fde0-a4ac-4375-aca8-e3122d747a30" containerName="maas-api" Sep 14 02:14:07.885066 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.885044 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:07.887469 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.887448 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7nzg8\"/\"openshift-service-ca.crt\"" Sep 14 02:14:07.887601 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.887451 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7nzg8\"/\"default-dockercfg-nvqmh\"" Sep 14 02:14:07.888424 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.888406 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7nzg8\"/\"kube-root-ca.crt\"" Sep 14 02:14:07.893939 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.893916 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj"] Sep 14 02:14:07.987383 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.987344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-lib-modules\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:07.987581 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.987392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-podres\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:07.987581 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.987431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pfvx\" (UniqueName: \"kubernetes.io/projected/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-kube-api-access-2pfvx\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:07.987581 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.987484 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-sys\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:07.987581 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:07.987547 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-proc\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088723 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088686 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-lib-modules\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088723 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088730 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-podres\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088958 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pfvx\" (UniqueName: \"kubernetes.io/projected/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-kube-api-access-2pfvx\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088958 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-sys\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088958 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088892 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-podres\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088958 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088901 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-lib-modules\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.088958 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088899 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-sys\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.089158 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.088969 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-proc\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.089158 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.089063 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-proc\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.096565 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.096546 2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pfvx\" (UniqueName: \"kubernetes.io/projected/a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75-kube-api-access-2pfvx\") pod \"perf-node-gather-daemonset-h5txj\" (UID: \"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75\") " pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.106906 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.106872 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-lf689_45f2ced9-74a7-4414-af03-eeb98cdd90d7/download-server/0.log" Sep 14 02:14:08.196568 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.196471 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:08.469093 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.469067 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj"] Sep 14 02:14:08.471802 ip-10-0-132-205 kubenswrapper[2573]: W0914 02:14:08.471772 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda4b280a6_8bd6_4f3b_8e8d_df6b4982ab75.slice/crio-033c3e619ecf2a951badde6298f018e818e84c319f3666017a95874bdd3f299c WatchSource:0}: Error finding container 033c3e619ecf2a951badde6298f018e818e84c319f3666017a95874bdd3f299c: Status 404 returned error can't find the container with id 033c3e619ecf2a951badde6298f018e818e84c319f3666017a95874bdd3f299c Sep 14 02:14:08.515051 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:08.515016 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" event={"ID":"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75","Type":"ContainerStarted","Data":"033c3e619ecf2a951badde6298f018e818e84c319f3666017a95874bdd3f299c"} Sep 14 02:14:09.524894 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.524857 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" event={"ID":"a4b280a6-8bd6-4f3b-8e8d-df6b4982ab75","Type":"ContainerStarted","Data":"92d6167ae9d8bbf0c8122ba4db5191b3a7968fc71b3c2f150c9b4d8eac85266c"} Sep 14 02:14:09.525304 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.524955 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:09.540737 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.540693 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" podStartSLOduration=2.540680753 podStartE2EDuration="2.540680753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:14:09.538765433 +0000 UTC m=+2596.211881608" watchObservedRunningTime="2026-09-14 02:14:09.540680753 +0000 UTC m=+2596.213796927" Sep 14 02:14:09.880347 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.880268 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vwkwr_bf444def-59a8-44d1-8ac4-8240cdec1922/dns/0.log" Sep 14 02:14:09.899712 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.899687 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vwkwr_bf444def-59a8-44d1-8ac4-8240cdec1922/kube-rbac-proxy/0.log" Sep 14 02:14:09.922408 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:09.922387 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6pflr_5206f48c-891c-4b88-b4b2-38c4cbc06a0b/dns-node-resolver/0.log" Sep 14 02:14:10.540083 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:10.540056 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kbv7x_00f4c9be-6791-4072-adfe-37b3b288e7e3/node-ca/0.log" Sep 14 02:14:12.131736 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:12.131711 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/0.log" Sep 14 02:14:12.134787 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:12.134764 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/1.log" Sep 14 02:14:12.194997 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:12.194969 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:14:13.109114 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:13.109061 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nk9kd_36788e14-cf89-4938-8495-fafa3349508a/serve-healthcheck-canary/0.log" Sep 14 02:14:13.709957 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:13.709904 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzl9s_9f4cd921-9291-452b-87ba-1e333d57050b/kube-rbac-proxy/0.log" Sep 14 02:14:13.731741 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:13.731713 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzl9s_9f4cd921-9291-452b-87ba-1e333d57050b/exporter/0.log" Sep 14 02:14:13.754428 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:13.754401 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzl9s_9f4cd921-9291-452b-87ba-1e333d57050b/extractor/0.log" Sep 14 02:14:15.539585 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:15.539512 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7nzg8/perf-node-gather-daemonset-h5txj" Sep 14 02:14:15.771028 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:15.771006 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lzzmz_aafa3390-8519-4cb7-8702-32dfdafbf073/manager/0.log" Sep 14 02:14:16.016650 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:16.016617 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gwc5w_f511c637-3e52-4bf2-9d4b-ebace521fecf/manager/0.log" Sep 14 02:14:16.064751 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:16.064721 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-59dfd7d9fd-lwkrw_89c66e52-cbfd-46ec-8f21-6f7837854bc0/manager/0.log" Sep 14 02:14:16.268589 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:16.268503 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-7xsrb_7b780b5e-a9f4-4672-8c15-e86b90c37ce6/manager/0.log" Sep 14 02:14:16.286177 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:16.286155 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c5d46b944-7xsrb_7b780b5e-a9f4-4672-8c15-e86b90c37ce6/copy-manifests/0.log" Sep 14 02:14:17.549077 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:17.549050 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f97d9f44d-46hst_56725921-913b-4ca7-a9db-5a92fd15a5e1/manager/0.log" Sep 14 02:14:18.473903 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:18.473880 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/0.log" Sep 14 02:14:18.476485 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:18.476469 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/1.log" Sep 14 02:14:18.527266 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:18.527175 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:14:20.890517 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:20.890482 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/0.log" Sep 14 02:14:20.893284 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:20.893234 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-cqcn9_b5732174-f596-4ad3-9d33-105200595126/istio-proxy/1.log" Sep 14 02:14:20.942385 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:20.942360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-c7wsb_b3e827c8-d189-4b67-ae62-6a60e9ce057a/payload-processing/0.log" Sep 14 02:14:24.565152 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.565122 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/kube-multus-additional-cni-plugins/0.log" Sep 14 02:14:24.586841 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.586814 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/egress-router-binary-copy/0.log" Sep 14 02:14:24.606927 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.606902 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/cni-plugins/0.log" Sep 14 02:14:24.626454 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.626434 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/bond-cni-plugin/0.log" Sep 14 02:14:24.646496 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.646476 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/routeoverride-cni/0.log" Sep 14 02:14:24.665477 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.665455 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/whereabouts-cni-bincopy/0.log" Sep 14 02:14:24.684230 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.684210 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqwl7_ee2b440b-a820-4ec4-aaaf-83ed2d4ca264/whereabouts-cni/0.log" Sep 14 02:14:24.979834 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:24.979805 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vgv5x_75f8117e-a6e9-40c5-9e40-06c899f78b3b/kube-multus/0.log" Sep 14 02:14:25.085514 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:25.085483 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zflq9_8cc55a86-c227-45a3-93ba-721ba72f5346/network-metrics-daemon/0.log" Sep 14 02:14:25.102324 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:25.102224 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zflq9_8cc55a86-c227-45a3-93ba-721ba72f5346/kube-rbac-proxy/0.log" Sep 14 02:14:26.074312 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.074289 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/ovn-controller/0.log" Sep 14 02:14:26.090990 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.090966 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/ovn-acl-logging/0.log" Sep 14 02:14:26.115417 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.115396 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/kube-rbac-proxy-node/0.log" Sep 14 02:14:26.134821 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.134797 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/kube-rbac-proxy-ovn-metrics/0.log" Sep 14 02:14:26.151723 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.151702 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/northd/0.log" Sep 14 02:14:26.171909 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.171886 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/nbdb/0.log" Sep 14 02:14:26.191204 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.191182 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/sbdb/0.log" Sep 14 02:14:26.245583 ip-10-0-132-205 kubenswrapper[2573]: I0914 02:14:26.245558 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kfb47_4a03df08-6a5b-4188-9ff8-c69b5ae0904e/ovnkube-controller/0.log"