Sep 11 12:12:16.502122 ip-10-0-140-239 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 12:12:16.502137 ip-10-0-140-239 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 12:12:16.502149 ip-10-0-140-239 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 12:12:16.502482 ip-10-0-140-239 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 12:12:26.527990 ip-10-0-140-239 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 12:12:26.528008 ip-10-0-140-239 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d790e5cf27a24bdf9b19f1bd1bb60904 -- Sep 11 12:14:18.897553 ip-10-0-140-239 systemd[1]: Starting Kubernetes Kubelet... Sep 11 12:14:19.255412 ip-10-0-140-239 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 12:14:19.255412 ip-10-0-140-239 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 12:14:19.255412 ip-10-0-140-239 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 12:14:19.255412 ip-10-0-140-239 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 12:14:19.255412 ip-10-0-140-239 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 12:14:19.257342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.257248 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 12:14:19.260798 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260784 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.260798 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260799 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260803 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260806 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260809 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260811 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260815 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260818 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260821 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260823 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260826 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260829 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260832 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260834 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260847 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260850 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260852 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260855 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260857 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260860 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260862 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.260862 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260865 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260867 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260870 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260873 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260876 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260878 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260881 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260883 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260886 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260888 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260890 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260893 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260895 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260897 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260900 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260902 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260905 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260909 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260913 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.261358 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260915 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260918 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260920 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260923 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260926 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260928 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260933 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260941 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260944 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260946 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260949 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260952 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260954 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260957 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260960 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260962 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260965 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260967 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260970 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260972 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.261858 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260975 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260977 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260980 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260982 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260985 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260987 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260989 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260992 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260994 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260996 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.260999 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261001 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261004 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261006 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261009 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261011 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261014 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261016 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261020 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261023 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.262359 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261026 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261029 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261031 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261034 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261038 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261041 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261044 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261047 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261050 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261053 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261055 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261058 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261062 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261065 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261068 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261071 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261073 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261076 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.261079 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.262839 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262575 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262582 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262585 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262589 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262592 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262594 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262597 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262601 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262603 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262606 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262609 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262611 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262614 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262616 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262619 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262622 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262626 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262628 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262631 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.263327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262634 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262637 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262640 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262642 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262647 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262650 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262653 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262656 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262659 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262662 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262665 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262667 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262670 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262672 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262674 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262677 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262679 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262682 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262685 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262687 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.263791 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262690 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262692 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262695 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262697 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262700 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262702 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262704 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262707 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262710 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262713 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262715 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262718 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262720 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262723 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262726 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262729 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262731 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262734 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262736 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262738 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.264357 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262741 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262743 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262746 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262749 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262752 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262755 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262758 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262761 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262764 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262767 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262770 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262773 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262776 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262778 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262781 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262784 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262786 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262789 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262791 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.264847 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262794 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262796 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262799 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262802 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262805 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262807 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262810 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262812 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262815 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262817 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262820 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262822 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262825 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262827 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262830 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262833 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262837 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262840 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262843 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262845 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.265327 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.262848 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263194 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263199 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263201 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263204 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263209 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263212 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263215 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263218 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263220 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263223 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263225 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263228 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263230 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263234 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263236 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263239 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263241 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263244 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.265820 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263246 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263249 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263251 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263254 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263256 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263259 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263261 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263266 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263268 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263286 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263289 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263291 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263294 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263296 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263299 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263302 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263304 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263306 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263309 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263311 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.266282 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263314 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263316 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263319 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263321 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263324 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263327 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263329 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263333 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263336 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263339 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263342 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263344 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263347 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263349 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263352 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263354 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263357 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263359 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263363 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.266770 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263367 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263369 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263372 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263375 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263378 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263380 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263383 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263385 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263388 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263390 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263393 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263395 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263397 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263400 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263402 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263405 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263407 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263410 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263412 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263414 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.267293 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263417 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263420 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263423 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263425 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263429 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263432 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263434 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263437 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263440 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263443 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263446 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263448 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263451 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263453 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263456 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263458 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263460 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263463 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263465 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.267975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263468 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263470 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.263473 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263542 2575 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263576 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263582 2575 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263586 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263590 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263593 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263597 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263602 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263605 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263608 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263611 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263614 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263617 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263621 2575 flags.go:64] FLAG: --cgroup-root="" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263624 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263626 2575 flags.go:64] FLAG: --client-ca-file="" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263629 2575 flags.go:64] FLAG: --cloud-provider="external" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263632 2575 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263636 2575 flags.go:64] FLAG: --cluster-domain="" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263638 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263641 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 12:14:19.268473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263644 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263647 2575 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263650 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263653 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263656 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263659 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263662 2575 flags.go:64] FLAG: --contention-profiling="false" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263664 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263667 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263670 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263673 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263676 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263680 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263683 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263686 2575 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263689 2575 flags.go:64] FLAG: --enable-server="true" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263691 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263695 2575 flags.go:64] FLAG: --event-burst="100" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263712 2575 flags.go:64] FLAG: --event-qps="50" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263716 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263719 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263722 2575 flags.go:64] FLAG: --eviction-hard="" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263730 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263733 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263736 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 12:14:19.269140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263739 2575 flags.go:64] FLAG: --eviction-soft="" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263742 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263745 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263748 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263751 2575 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263753 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263756 2575 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263759 2575 flags.go:64] FLAG: --feature-gates="" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263762 2575 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263765 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263768 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263771 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263774 2575 flags.go:64] FLAG: --healthz-port="10248" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263776 2575 flags.go:64] FLAG: --help="false" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263779 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263782 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263784 2575 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263787 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263790 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263795 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263798 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263800 2575 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263803 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 12:14:19.269774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263806 2575 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263808 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263811 2575 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263814 2575 flags.go:64] FLAG: --kube-reserved="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263817 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263819 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263822 2575 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263825 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263827 2575 flags.go:64] FLAG: --lock-file="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263830 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263833 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263836 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263841 2575 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263844 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263846 2575 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263849 2575 flags.go:64] FLAG: --logging-format="text" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263853 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263856 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263858 2575 flags.go:64] FLAG: --manifest-url="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263861 2575 flags.go:64] FLAG: --manifest-url-header="" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263865 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263868 2575 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263872 2575 flags.go:64] FLAG: --max-pods="110" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263874 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263877 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 12:14:19.270417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263880 2575 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263882 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263885 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263889 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263892 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263898 2575 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263901 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263903 2575 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263906 2575 flags.go:64] FLAG: --pod-cidr="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263909 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263913 2575 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263916 2575 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263919 2575 flags.go:64] FLAG: --pods-per-core="0" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263922 2575 flags.go:64] FLAG: --port="10250" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263924 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263927 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b31d790aefd00dc5" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263930 2575 flags.go:64] FLAG: --qos-reserved="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263933 2575 flags.go:64] FLAG: --read-only-port="10255" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263937 2575 flags.go:64] FLAG: --register-node="true" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263939 2575 flags.go:64] FLAG: --register-with-taints="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263943 2575 flags.go:64] FLAG: --registry-burst="10" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263946 2575 flags.go:64] FLAG: --registry-qps="5" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263948 2575 flags.go:64] FLAG: --reserved-cpus="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263952 2575 flags.go:64] FLAG: --reserved-memory="" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263955 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 12:14:19.271032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263958 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263961 2575 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263964 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263967 2575 flags.go:64] FLAG: --runonce="false" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263969 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263972 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263975 2575 flags.go:64] FLAG: --seccomp-default="false" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263977 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263980 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263984 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263987 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263991 2575 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263994 2575 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263996 2575 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.263999 2575 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264002 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264004 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264007 2575 flags.go:64] FLAG: --system-cgroups="" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264010 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264015 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264017 2575 flags.go:64] FLAG: --tls-cert-file="" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264020 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264024 2575 flags.go:64] FLAG: --tls-min-version="" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264027 2575 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264029 2575 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 12:14:19.271650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264032 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264035 2575 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264038 2575 flags.go:64] FLAG: --v="2" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264042 2575 flags.go:64] FLAG: --version="false" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264045 2575 flags.go:64] FLAG: --vmodule="" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264050 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.264053 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264168 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264172 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264175 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264178 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264181 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264184 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264187 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264190 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264193 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264197 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264200 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264204 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264207 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264210 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.272286 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264213 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264216 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264218 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264220 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264223 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264235 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264239 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264242 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264244 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264247 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264249 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264252 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264255 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264258 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264260 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264263 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264267 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264287 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264291 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264296 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.272814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264300 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264304 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264307 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264309 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264312 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264315 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264317 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264320 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264322 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264327 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264331 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264334 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264337 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264340 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264342 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264345 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264347 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264350 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264353 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.273342 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264356 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264360 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264363 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264366 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264368 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264371 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264374 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264377 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264380 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264384 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264387 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264390 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264392 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264395 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264397 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264399 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264402 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264404 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264407 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264409 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.273828 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264412 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264414 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264417 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264420 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264423 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264425 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264428 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264430 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264433 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264435 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264438 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264440 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264442 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264445 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264447 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264450 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264452 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264455 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264457 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.274332 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264460 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264462 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264466 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264469 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264471 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264473 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.264476 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.265075 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.274742 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 12:14:19.274794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.274756 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 12:14:19.275380 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.275369 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 12:14:19.275415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.275380 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 12:14:19.275446 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275435 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.275446 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275440 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.275446 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275444 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.275446 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275447 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275451 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275454 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275457 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275460 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275462 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275466 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275468 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275471 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275473 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275476 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275479 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275481 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275484 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275486 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275489 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275492 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275494 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275497 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275500 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.275555 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275502 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275505 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275507 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275510 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275512 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275517 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275520 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275524 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275528 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275532 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275535 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275538 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275541 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275544 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275547 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275550 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275553 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275556 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275559 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275562 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.276033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275564 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275567 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275570 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275572 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275575 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275577 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275580 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275582 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275585 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275587 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275590 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275593 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275595 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275598 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275600 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275603 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275605 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275608 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275610 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275613 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.276549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275615 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275617 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275620 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275622 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275626 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275630 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275634 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275637 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275640 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275643 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275646 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275649 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275652 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275654 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275657 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275660 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275663 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275666 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275669 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275671 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.277041 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275674 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275676 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275679 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275681 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275684 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275686 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275689 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275693 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275696 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275698 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275701 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275703 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275706 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275708 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275711 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.277620 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275713 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.275718 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275820 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275826 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275829 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275832 2575 feature_gate.go:328] unrecognized feature gate: Example Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275835 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275837 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275840 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275842 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275845 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275847 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275850 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275852 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275855 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 12:14:19.277985 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275858 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275860 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275863 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275865 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275868 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275873 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275876 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275878 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275881 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275884 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275886 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275888 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275891 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275893 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275896 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275898 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275900 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275903 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275905 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275908 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 12:14:19.278366 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275911 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275913 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275916 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275919 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275921 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275924 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275927 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275929 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275932 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275934 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275936 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275939 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275942 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275945 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275947 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275949 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275952 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275955 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275957 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275959 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 12:14:19.278846 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275962 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275965 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275967 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275970 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275972 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275976 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275979 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275981 2575 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275984 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275987 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275989 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275992 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275994 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.275997 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276000 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276002 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276005 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276007 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276011 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 12:14:19.279341 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276014 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276017 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276020 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276022 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276025 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276027 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276031 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276034 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276037 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276040 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276042 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276045 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276048 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276050 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276052 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276055 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276057 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276060 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276062 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276064 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 12:14:19.279808 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276067 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276069 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276072 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276074 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276077 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276079 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:19.276082 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.276087 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.276183 2575 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 12:14:19.280308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.279549 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 12:14:19.280535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.280319 2575 server.go:1019] "Starting client certificate rotation" Sep 11 12:14:19.280535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.280424 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 12:14:19.280535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.280458 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 12:14:19.283153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.283139 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 12:14:19.286498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.286482 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 12:14:19.295773 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.295683 2575 log.go:25] "Validated CRI v1 runtime API" Sep 11 12:14:19.300093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.300075 2575 log.go:25] "Validated CRI v1 image API" Sep 11 12:14:19.301323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.301191 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 12:14:19.306076 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.306052 2575 fs.go:135] Filesystem UUIDs: map[3ae67e22-0437-4f28-bd94-68b75c2ae3e3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d9f1c3c2-198c-46b4-ac1b-059740a7ec80:/dev/nvme0n1p3] Sep 11 12:14:19.306153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.306074 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 12:14:19.310743 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.310642 2575 manager.go:216] Machine: {Timestamp:2026-09-11 12:14:19.309493336 +0000 UTC m=+0.308035678 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099142 MemoryCapacity:32812142592 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ece2806d198d6f91db219e02dae3 SystemUUID:ec21ece2-806d-198d-6f91-db219e02dae3 BootID:d790e5cf-27a2-4bdf-9b19-f1bd1bb60904 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406069248 Type:vfs Inodes:4005388 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:3b:97:1b:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:3b:97:1b:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:99:fc:11:ea:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812142592 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 12:14:19.310743 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.310736 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 12:14:19.310851 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.310803 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 12:14:19.311684 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.311660 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 12:14:19.311808 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.311686 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-239.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 12:14:19.312578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.312567 2575 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 12:14:19.312610 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.312581 2575 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 12:14:19.312610 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.312588 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 12:14:19.312772 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.312758 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 12:14:19.313142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.313132 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 12:14:19.313776 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.313766 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 12:14:19.317287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317262 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 11 12:14:19.317392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317382 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 12:14:19.317427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317409 2575 kubelet.go:486] "Attempting to sync node with API server" Sep 11 12:14:19.317427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317418 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 12:14:19.317481 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317431 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 12:14:19.317481 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317440 2575 kubelet.go:398] "Adding apiserver pod source" Sep 11 12:14:19.317481 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317449 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 12:14:19.317803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317789 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 12:14:19.317847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.317819 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 12:14:19.320916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.320901 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 12:14:19.322918 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.322905 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 12:14:19.322968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.322934 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 12:14:19.325348 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325336 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325353 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325359 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325364 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325372 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325377 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 12:14:19.325390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325390 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 12:14:19.325543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325397 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 12:14:19.325543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325403 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 12:14:19.325543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325408 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 12:14:19.325543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325421 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 12:14:19.325543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.325426 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 12:14:19.326586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.326577 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 12:14:19.326586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.326586 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 12:14:19.330014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.330000 2575 server.go:1268] "Started kubelet" Sep 11 12:14:19.330114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.330077 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 12:14:19.330214 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.330088 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 12:14:19.330214 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.330148 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 12:14:19.330767 ip-10-0-140-239 systemd[1]: Started Kubernetes Kubelet. Sep 11 12:14:19.331654 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.331638 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 12:14:19.332672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.332658 2575 server.go:310] "Adding debug handlers to kubelet server" Sep 11 12:14:19.334129 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.334105 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 12:14:19.337671 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.337650 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 12:14:19.338114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338096 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 12:14:19.338334 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.338312 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.338715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338569 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 12:14:19.338715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338579 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 12:14:19.338715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338586 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 12:14:19.338715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338683 2575 reconstruct.go:97] "Volume reconstruction finished" Sep 11 12:14:19.338715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.338691 2575 reconciler.go:29] "Reconciler: start to sync state" Sep 11 12:14:19.340020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340005 2575 factory.go:55] Registering systemd factory Sep 11 12:14:19.340103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340025 2575 factory.go:223] Registration of the systemd container factory successfully Sep 11 12:14:19.340222 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340211 2575 factory.go:153] Registering CRI-O factory Sep 11 12:14:19.340291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340224 2575 factory.go:223] Registration of the crio container factory successfully Sep 11 12:14:19.340291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340289 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 12:14:19.340358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340304 2575 factory.go:103] Registering Raw factory Sep 11 12:14:19.340358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340314 2575 manager.go:1196] Started watching for new ooms in manager Sep 11 12:14:19.340706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.340696 2575 manager.go:349] Starting recovery of all containers Sep 11 12:14:19.343771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.343749 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-239.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 12:14:19.344339 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.344311 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 12:14:19.344944 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.344920 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 12:14:19.345036 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.344945 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 12:14:19.345432 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.345411 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-239.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 12:14:19.346013 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.344977 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e5c33b3b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.329968955 +0000 UTC m=+0.328511300,LastTimestamp:2026-09-11 12:14:19.329968955 +0000 UTC m=+0.328511300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.350458 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.350443 2575 manager.go:354] Recovery completed Sep 11 12:14:19.351698 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.351641 2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 12:14:19.354497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.354485 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.356121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356093 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.356172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356134 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.356172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356148 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.356690 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356666 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 12:14:19.356690 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356681 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 12:14:19.356815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356698 2575 state_mem.go:36] "Initialized new in-memory state store" Sep 11 12:14:19.356815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.356714 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.357395 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.357381 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.357448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.357406 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.357448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.357419 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.358709 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.358695 2575 policy_none.go:49] "None policy: Start" Sep 11 12:14:19.358768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.358714 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 12:14:19.358768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.358727 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 12:14:19.358868 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.358725 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e7524689 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356120713 +0000 UTC m=+0.354663054,LastTimestamp:2026-09-11 12:14:19.356120713 +0000 UTC m=+0.354663054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.360725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.360711 2575 policy_none.go:47] "Start" Sep 11 12:14:19.372855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.372774 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e75297d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356141529 +0000 UTC m=+0.354683871,LastTimestamp:2026-09-11 12:14:19.356141529 +0000 UTC m=+0.354683871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.376321 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.376244 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e752c0e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356152035 +0000 UTC m=+0.354694378,LastTimestamp:2026-09-11 12:14:19.356152035 +0000 UTC m=+0.354694378,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.383168 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.383079 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-239.ec2.internal.18d44337e7524689\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e7524689 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356120713 +0000 UTC m=+0.354663054,LastTimestamp:2026-09-11 12:14:19.357394113 +0000 UTC m=+0.355936455,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.396816 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.396800 2575 manager.go:337] "Starting Device Plugin manager" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.396840 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.396854 2575 server.go:87] "Starting device plugin registration server" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.397130 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.397140 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.397221 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.397321 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.397328 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.397755 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.397808 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.400559 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6v5l" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.405877 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-239.ec2.internal.18d44337e75297d9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e75297d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356141529 +0000 UTC m=+0.354683871,LastTimestamp:2026-09-11 12:14:19.357413106 +0000 UTC m=+0.355955448,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.408914 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l6v5l" Sep 11 12:14:19.420016 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.415158 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-239.ec2.internal.18d44337e752c0e3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-239.ec2.internal.18d44337e752c0e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-239.ec2.internal,UID:ip-10-0-140-239.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-239.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-239.ec2.internal,},FirstTimestamp:2026-09-11 12:14:19.356152035 +0000 UTC m=+0.354694378,LastTimestamp:2026-09-11 12:14:19.357422864 +0000 UTC m=+0.355965205,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-239.ec2.internal,}" Sep 11 12:14:19.434625 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.434601 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 12:14:19.435866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.435843 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 12:14:19.435934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.435895 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 12:14:19.435934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.435917 2575 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 12:14:19.436009 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.435957 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 12:14:19.439092 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.439076 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 12:14:19.498178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.498157 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.498970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.498957 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.499039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.498984 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.499039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.498994 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.499039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.499018 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.507805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.507746 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.507805 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.507766 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-239.ec2.internal\": node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.536483 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.536461 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal"] Sep 11 12:14:19.536573 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.536543 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.537228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.537214 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.537336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.537243 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.537336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.537254 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.539401 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.539388 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.539539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.539526 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.539596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.539551 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.540037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540020 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.540114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540042 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.540114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540053 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.540114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540020 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.540207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540121 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.540207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.540138 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.542263 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.542248 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.542324 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.542297 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 12:14:19.542912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.542894 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientMemory" Sep 11 12:14:19.542989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.542916 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 12:14:19.542989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.542930 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeHasSufficientPID" Sep 11 12:14:19.557518 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.557500 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.577883 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.577860 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-239.ec2.internal\" not found" node="ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.582165 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.582151 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-239.ec2.internal\" not found" node="ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.639728 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.639697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.639812 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.639733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.657955 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.657935 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.740765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.740733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.740765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.740769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66903a53d0aed29fd4572d053e31c420-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"66903a53d0aed29fd4572d053e31c420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.740888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.740790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.740888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.740859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.740952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.740868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7d3bac48cef4011d55130ca1dfcfcce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal\" (UID: \"f7d3bac48cef4011d55130ca1dfcfcce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.758890 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.758838 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.841493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.841461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66903a53d0aed29fd4572d053e31c420-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"66903a53d0aed29fd4572d053e31c420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.841590 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.841512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66903a53d0aed29fd4572d053e31c420-config\") pod \"kube-apiserver-proxy-ip-10-0-140-239.ec2.internal\" (UID: \"66903a53d0aed29fd4572d053e31c420\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.859592 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.859571 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:19.879735 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.879719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.884867 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:19.884663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:19.960602 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:19.960576 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:20.061212 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.061160 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:20.161631 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.161605 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-239.ec2.internal\" not found" Sep 11 12:14:20.177455 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.177430 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 12:14:20.238639 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.238611 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" Sep 11 12:14:20.255783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.255764 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 12:14:20.257260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.257244 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" Sep 11 12:14:20.271149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.271124 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 12:14:20.280283 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.280253 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 12:14:20.280410 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.280391 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 12:14:20.318324 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.318263 2575 apiserver.go:52] "Watching apiserver" Sep 11 12:14:20.327347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.327318 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 12:14:20.327720 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.327698 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mznf6","openshift-image-registry/node-ca-w4tw4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal","openshift-multus/network-metrics-daemon-9mtmd","openshift-network-diagnostics/network-check-target-g6vhr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v","openshift-dns/node-resolver-5wrfm","openshift-multus/multus-86dcp","openshift-multus/multus-additional-cni-plugins-jgnzl","openshift-network-operator/iptables-alerter-6h22t","openshift-ovn-kubernetes/ovnkube-node-wqf8h","kube-system/global-pull-secret-syncer-glbfl","kube-system/konnectivity-agent-vbtsc"] Sep 11 12:14:20.330132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.330115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.332191 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.332171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.332900 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.332877 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pd8gb\"" Sep 11 12:14:20.332984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.332935 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.332984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.332949 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.335713 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.335685 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.335809 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.335772 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 12:14:20.337891 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.335865 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.337891 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.336117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.337891 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.336403 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w9vbg\"" Sep 11 12:14:20.337891 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.336619 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:20.338133 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.337880 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 12:14:20.338788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.338750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:20.338888 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.338840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:20.341061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.341042 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.343236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.343219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.343989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.343968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.343999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-var-lib-kubelet\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-tuned\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-run\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-tmp\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9qg\" (UniqueName: \"kubernetes.io/projected/c58c631d-96c5-4da1-b807-41709d259e3b-kube-api-access-5b9qg\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysconfig\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn5cp\" (UniqueName: \"kubernetes.io/projected/0ba65492-6ff4-489f-b229-3fe6896728aa-kube-api-access-pn5cp\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.344367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:20.344367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-kubernetes\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ba65492-6ff4-489f-b229-3fe6896728aa-serviceca\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkz6q\" (UniqueName: \"kubernetes.io/projected/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-kube-api-access-nkz6q\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-conf\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-lib-modules\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-host\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba65492-6ff4-489f-b229-3fe6896728aa-host\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.344603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-device-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344610 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-modprobe-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-systemd\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-sys\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344698 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/02643625-d215-4959-a4b6-2cd85e9ad9a2-kube-api-access-rmgm9\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.344883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344844 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.345099 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.344901 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.345358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.345345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.345895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.345874 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 12:14:20.345895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.345890 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.346153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.346137 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lrgl6\"" Sep 11 12:14:20.346208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.346190 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h85m7\"" Sep 11 12:14:20.347903 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.347883 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.348052 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.348034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.349144 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.349123 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 12:14:20.349219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.349181 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 12:14:20.349446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.349428 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2r7jt\"" Sep 11 12:14:20.350558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.350535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.350691 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.350675 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 12:14:20.350767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.350688 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 12:14:20.350767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.350676 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.351389 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.351371 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvqds\"" Sep 11 12:14:20.351482 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.351432 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.352988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.352969 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.353499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.353482 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs9r5\"" Sep 11 12:14:20.353583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.353560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 12:14:20.354423 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.354407 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.354487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.354429 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.355267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.355247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.355373 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.355324 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:20.357502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.357487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.358256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.358240 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 12:14:20.358595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.358579 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 12:14:20.358725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.358678 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 12:14:20.358795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.358758 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 12:14:20.359009 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.358987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n6hql\"" Sep 11 12:14:20.359993 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.359977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 12:14:20.361153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.361134 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 12:14:20.361153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.361149 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 12:14:20.361340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.361221 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 12:14:20.362065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.361692 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-frznq\"" Sep 11 12:14:20.362065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.361735 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 12:14:20.383943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.383923 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmwq5" Sep 11 12:14:20.394400 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.394370 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rmwq5" Sep 11 12:14:20.414243 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.414197 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 12:09:20 +0000 UTC" deadline="2028-03-14 19:34:22.709751674 +0000 UTC" Sep 11 12:14:20.414243 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.414241 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13207h20m2.295513726s" Sep 11 12:14:20.439252 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.439230 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 12:14:20.444956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.444934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-conf\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.444972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-lib-modules\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba65492-6ff4-489f-b229-3fe6896728aa-host\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.445042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-kubelet-config\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-hostroot\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-conf\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-node-log\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-lib-modules\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-sys\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ba65492-6ff4-489f-b229-3fe6896728aa-host\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.445154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445173 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-sys\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-sys-fs\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-socket-dir-parent\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-netns\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-script-lib\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-multus\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-kubelet\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-etc-kubernetes\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-var-lib-kubelet\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-var-lib-kubelet\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmf75\" (UniqueName: \"kubernetes.io/projected/a073adff-9215-4929-8b7a-4ba3b3522f50-kube-api-access-rmf75\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-kubelet\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445533 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b34698e3-de0b-4d89-8c31-901d9efea428-hosts-file\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cnibin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-k8s-cni-cncf-io\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-systemd-units\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-etc-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-log-socket\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjm7\" (UniqueName: \"kubernetes.io/projected/47381769-db6f-4f64-be82-c0561af5ec74-kube-api-access-jwjm7\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.445756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lp7\" (UniqueName: \"kubernetes.io/projected/a236ce12-2ee8-42a3-9dc6-82c584e91f24-kube-api-access-m5lp7\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-env-overrides\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ba65492-6ff4-489f-b229-3fe6896728aa-serviceca\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkz6q\" (UniqueName: \"kubernetes.io/projected/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-kube-api-access-nkz6q\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-system-cni-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmfdj\" (UniqueName: \"kubernetes.io/projected/b6854465-c2ab-44f6-bd2b-cc6375df2348-kube-api-access-rmfdj\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-slash\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-host\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-device-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-host\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.445992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-netns\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-device-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47381769-db6f-4f64-be82-c0561af5ec74-ovn-node-metrics-cert\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-modprobe-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-systemd\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-system-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-daemon-config\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-modprobe-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-multus-certs\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-systemd\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a073adff-9215-4929-8b7a-4ba3b3522f50-host-slash\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ba65492-6ff4-489f-b229-3fe6896728aa-serviceca\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.446205 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/02643625-d215-4959-a4b6-2cd85e9ad9a2-kube-api-access-rmgm9\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-bin\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.446322 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:20.946295009 +0000 UTC m=+1.944837337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34698e3-de0b-4d89-8c31-901d9efea428-tmp-dir\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.446804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-bin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a073adff-9215-4929-8b7a-4ba3b3522f50-iptables-alerter-script\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-netd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-registration-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/819dccc0-085c-40a5-b00c-80b876a761db-konnectivity-ca\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysctl-d\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446571 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-tuned\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-cnibin\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-dbus\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-systemd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/819dccc0-085c-40a5-b00c-80b876a761db-agent-certs\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-run\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-tmp\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9qg\" (UniqueName: \"kubernetes.io/projected/c58c631d-96c5-4da1-b807-41709d259e3b-kube-api-access-5b9qg\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-run\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.447518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxzs\" (UniqueName: \"kubernetes.io/projected/b34698e3-de0b-4d89-8c31-901d9efea428-kube-api-access-cxxzs\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cni-binary-copy\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-ovn\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-config\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysconfig\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn5cp\" (UniqueName: \"kubernetes.io/projected/0ba65492-6ff4-489f-b229-3fe6896728aa-kube-api-access-pn5cp\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446950 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-conf-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.446997 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-kubernetes\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.447020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.447043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-var-lib-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.447149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-sysconfig\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.447291 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-kubernetes\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.448038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.447341 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c58c631d-96c5-4da1-b807-41709d259e3b-socket-dir\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.449882 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.449863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-etc-tuned\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.449952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.449890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02643625-d215-4959-a4b6-2cd85e9ad9a2-tmp\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.454965 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.454948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkz6q\" (UniqueName: \"kubernetes.io/projected/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-kube-api-access-nkz6q\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.458178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.458162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9qg\" (UniqueName: \"kubernetes.io/projected/c58c631d-96c5-4da1-b807-41709d259e3b-kube-api-access-5b9qg\") pod \"aws-ebs-csi-driver-node-q9d4v\" (UID: \"c58c631d-96c5-4da1-b807-41709d259e3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.463298 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.463258 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:20.463378 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.463304 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:20.463378 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.463319 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:20.463456 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.463405 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:20.96338852 +0000 UTC m=+1.961930863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:20.464462 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.464439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgm9\" (UniqueName: \"kubernetes.io/projected/02643625-d215-4959-a4b6-2cd85e9ad9a2-kube-api-access-rmgm9\") pod \"tuned-mznf6\" (UID: \"02643625-d215-4959-a4b6-2cd85e9ad9a2\") " pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.465010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.464988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn5cp\" (UniqueName: \"kubernetes.io/projected/0ba65492-6ff4-489f-b229-3fe6896728aa-kube-api-access-pn5cp\") pod \"node-ca-w4tw4\" (UID: \"0ba65492-6ff4-489f-b229-3fe6896728aa\") " pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.519231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.519089 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-cnibin\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-dbus\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-systemd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-cnibin\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/819dccc0-085c-40a5-b00c-80b876a761db-agent-certs\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxzs\" (UniqueName: \"kubernetes.io/projected/b34698e3-de0b-4d89-8c31-901d9efea428-kube-api-access-cxxzs\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cni-binary-copy\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-ovn\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-config\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-conf-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-var-lib-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-kubelet-config\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-hostroot\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-node-log\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.548358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-conf-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548342 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-kubelet-config\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-hostroot\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548396 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-var-lib-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-binary-copy\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.548966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-systemd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-ovn\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.549183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-node-log\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cni-binary-copy\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-config\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.547876 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b2b3ef44-c676-4dbe-bd64-17e7a276afee-dbus\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-socket-dir-parent\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-netns\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-script-lib\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-multus\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6854465-c2ab-44f6-bd2b-cc6375df2348-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.549998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-kubelet\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-etc-kubernetes\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-socket-dir-parent\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmf75\" (UniqueName: \"kubernetes.io/projected/a073adff-9215-4929-8b7a-4ba3b3522f50-kube-api-access-rmf75\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-netns\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-kubelet\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-etc-kubernetes\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.550560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-kubelet\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-multus\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b34698e3-de0b-4d89-8c31-901d9efea428-hosts-file\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550223 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b34698e3-de0b-4d89-8c31-901d9efea428-hosts-file\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-kubelet\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cnibin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-k8s-cni-cncf-io\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-systemd-units\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-etc-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-cnibin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-etc-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-log-socket\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-k8s-cni-cncf-io\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550788 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-log-socket\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-run-openvswitch\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/819dccc0-085c-40a5-b00c-80b876a761db-agent-certs\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjm7\" (UniqueName: \"kubernetes.io/projected/47381769-db6f-4f64-be82-c0561af5ec74-kube-api-access-jwjm7\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.551399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lp7\" (UniqueName: \"kubernetes.io/projected/a236ce12-2ee8-42a3-9dc6-82c584e91f24-kube-api-access-m5lp7\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.550980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-systemd-units\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-env-overrides\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-system-cni-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmfdj\" (UniqueName: \"kubernetes.io/projected/b6854465-c2ab-44f6-bd2b-cc6375df2348-kube-api-access-rmfdj\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-slash\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-netns\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47381769-db6f-4f64-be82-c0561af5ec74-ovn-node-metrics-cert\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6854465-c2ab-44f6-bd2b-cc6375df2348-system-cni-dir\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-system-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-daemon-config\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-multus-certs\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-multus-certs\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-system-cni-dir\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.552244 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.551615 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-run-netns\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a073adff-9215-4929-8b7a-4ba3b3522f50-host-slash\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-bin\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.551970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-ovnkube-script-lib\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.552211 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:21.052185586 +0000 UTC m=+2.050727928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-run-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a236ce12-2ee8-42a3-9dc6-82c584e91f24-multus-daemon-config\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-slash\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34698e3-de0b-4d89-8c31-901d9efea428-tmp-dir\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-bin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a073adff-9215-4929-8b7a-4ba3b3522f50-iptables-alerter-script\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47381769-db6f-4f64-be82-c0561af5ec74-env-overrides\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-netd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552528 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a236ce12-2ee8-42a3-9dc6-82c584e91f24-host-var-lib-cni-bin\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-netd\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/819dccc0-085c-40a5-b00c-80b876a761db-konnectivity-ca\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a073adff-9215-4929-8b7a-4ba3b3522f50-host-slash\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.552764 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47381769-db6f-4f64-be82-c0561af5ec74-host-cni-bin\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.553032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/819dccc0-085c-40a5-b00c-80b876a761db-konnectivity-ca\") pod \"konnectivity-agent-vbtsc\" (UID: \"819dccc0-085c-40a5-b00c-80b876a761db\") " pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.553051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a073adff-9215-4929-8b7a-4ba3b3522f50-iptables-alerter-script\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.553730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.553211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34698e3-de0b-4d89-8c31-901d9efea428-tmp-dir\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.554030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.553982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47381769-db6f-4f64-be82-c0561af5ec74-ovn-node-metrics-cert\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.563771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.563748 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxzs\" (UniqueName: \"kubernetes.io/projected/b34698e3-de0b-4d89-8c31-901d9efea428-kube-api-access-cxxzs\") pod \"node-resolver-5wrfm\" (UID: \"b34698e3-de0b-4d89-8c31-901d9efea428\") " pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.566383 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.566360 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d3bac48cef4011d55130ca1dfcfcce.slice/crio-f397d133c70c54307f62bd4f012d8155f68efde76b1534f0ddf56eadb9270f00 WatchSource:0}: Error finding container f397d133c70c54307f62bd4f012d8155f68efde76b1534f0ddf56eadb9270f00: Status 404 returned error can't find the container with id f397d133c70c54307f62bd4f012d8155f68efde76b1534f0ddf56eadb9270f00 Sep 11 12:14:20.569308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.569242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmf75\" (UniqueName: \"kubernetes.io/projected/a073adff-9215-4929-8b7a-4ba3b3522f50-kube-api-access-rmf75\") pod \"iptables-alerter-6h22t\" (UID: \"a073adff-9215-4929-8b7a-4ba3b3522f50\") " pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.569392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.569339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjm7\" (UniqueName: \"kubernetes.io/projected/47381769-db6f-4f64-be82-c0561af5ec74-kube-api-access-jwjm7\") pod \"ovnkube-node-wqf8h\" (UID: \"47381769-db6f-4f64-be82-c0561af5ec74\") " pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.569708 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.569689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmfdj\" (UniqueName: \"kubernetes.io/projected/b6854465-c2ab-44f6-bd2b-cc6375df2348-kube-api-access-rmfdj\") pod \"multus-additional-cni-plugins-jgnzl\" (UID: \"b6854465-c2ab-44f6-bd2b-cc6375df2348\") " pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.569961 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.569943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lp7\" (UniqueName: \"kubernetes.io/projected/a236ce12-2ee8-42a3-9dc6-82c584e91f24-kube-api-access-m5lp7\") pod \"multus-86dcp\" (UID: \"a236ce12-2ee8-42a3-9dc6-82c584e91f24\") " pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.571410 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.571397 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:14:20.635535 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.635518 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66903a53d0aed29fd4572d053e31c420.slice/crio-bf3b11b9d1cd4a39203f8aeee65c549eb7dd6b670031a328cf47dbb220579fb6 WatchSource:0}: Error finding container bf3b11b9d1cd4a39203f8aeee65c549eb7dd6b670031a328cf47dbb220579fb6: Status 404 returned error can't find the container with id bf3b11b9d1cd4a39203f8aeee65c549eb7dd6b670031a328cf47dbb220579fb6 Sep 11 12:14:20.659865 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.659833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mznf6" Sep 11 12:14:20.666929 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.666908 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02643625_d215_4959_a4b6_2cd85e9ad9a2.slice/crio-ebb12cfcde0d6f5364f3d0ffc4815b07139fc98436152793a127427a44f5791a WatchSource:0}: Error finding container ebb12cfcde0d6f5364f3d0ffc4815b07139fc98436152793a127427a44f5791a: Status 404 returned error can't find the container with id ebb12cfcde0d6f5364f3d0ffc4815b07139fc98436152793a127427a44f5791a Sep 11 12:14:20.680345 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.680327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w4tw4" Sep 11 12:14:20.687730 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.687708 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba65492_6ff4_489f_b229_3fe6896728aa.slice/crio-50e68e7d68426e22be1b8cbf995eeca02b73c20e6a39850e1da7dd914455b9bf WatchSource:0}: Error finding container 50e68e7d68426e22be1b8cbf995eeca02b73c20e6a39850e1da7dd914455b9bf: Status 404 returned error can't find the container with id 50e68e7d68426e22be1b8cbf995eeca02b73c20e6a39850e1da7dd914455b9bf Sep 11 12:14:20.695538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.695520 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" Sep 11 12:14:20.699173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.699156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5wrfm" Sep 11 12:14:20.703172 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.703155 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58c631d_96c5_4da1_b807_41709d259e3b.slice/crio-4eee0fade97a0c251070b5970ecd8d774fe52b18015f358c9fd628316e4a8906 WatchSource:0}: Error finding container 4eee0fade97a0c251070b5970ecd8d774fe52b18015f358c9fd628316e4a8906: Status 404 returned error can't find the container with id 4eee0fade97a0c251070b5970ecd8d774fe52b18015f358c9fd628316e4a8906 Sep 11 12:14:20.707712 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.707692 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34698e3_de0b_4d89_8c31_901d9efea428.slice/crio-ac7d5ddb5ae205b2a1ad6d86d1e0b31808490051659ebd407c3a276907c01ecb WatchSource:0}: Error finding container ac7d5ddb5ae205b2a1ad6d86d1e0b31808490051659ebd407c3a276907c01ecb: Status 404 returned error can't find the container with id ac7d5ddb5ae205b2a1ad6d86d1e0b31808490051659ebd407c3a276907c01ecb Sep 11 12:14:20.712568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.712553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86dcp" Sep 11 12:14:20.722151 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.722131 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda236ce12_2ee8_42a3_9dc6_82c584e91f24.slice/crio-aad8c480749eae2d23d9fb18eb74d341c80728e74f65388019316a3584846526 WatchSource:0}: Error finding container aad8c480749eae2d23d9fb18eb74d341c80728e74f65388019316a3584846526: Status 404 returned error can't find the container with id aad8c480749eae2d23d9fb18eb74d341c80728e74f65388019316a3584846526 Sep 11 12:14:20.736236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.736211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" Sep 11 12:14:20.743229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.743212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6h22t" Sep 11 12:14:20.743737 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.743721 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6854465_c2ab_44f6_bd2b_cc6375df2348.slice/crio-e971691aba4450a0e83442b83d80a9d78167f7733c1c7f9be239b569b12f27ac WatchSource:0}: Error finding container e971691aba4450a0e83442b83d80a9d78167f7733c1c7f9be239b569b12f27ac: Status 404 returned error can't find the container with id e971691aba4450a0e83442b83d80a9d78167f7733c1c7f9be239b569b12f27ac Sep 11 12:14:20.750701 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.750684 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda073adff_9215_4929_8b7a_4ba3b3522f50.slice/crio-31a660dab39ac3100e7df715eda37313cf608236d82feeaeb72b90b5e8977faf WatchSource:0}: Error finding container 31a660dab39ac3100e7df715eda37313cf608236d82feeaeb72b90b5e8977faf: Status 404 returned error can't find the container with id 31a660dab39ac3100e7df715eda37313cf608236d82feeaeb72b90b5e8977faf Sep 11 12:14:20.769754 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.769738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:20.775340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.775321 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:20.778168 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.778128 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47381769_db6f_4f64_be82_c0561af5ec74.slice/crio-de84f38308edebd4813e8bd07878fae2366162c4ad7a30bb93ac288faa1cad5f WatchSource:0}: Error finding container de84f38308edebd4813e8bd07878fae2366162c4ad7a30bb93ac288faa1cad5f: Status 404 returned error can't find the container with id de84f38308edebd4813e8bd07878fae2366162c4ad7a30bb93ac288faa1cad5f Sep 11 12:14:20.783742 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:20.783725 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819dccc0_085c_40a5_b00c_80b876a761db.slice/crio-6c57326bafc26a0f2328775d64cd69fb37998f6e998ed7e1908f6bde45b0ad25 WatchSource:0}: Error finding container 6c57326bafc26a0f2328775d64cd69fb37998f6e998ed7e1908f6bde45b0ad25: Status 404 returned error can't find the container with id 6c57326bafc26a0f2328775d64cd69fb37998f6e998ed7e1908f6bde45b0ad25 Sep 11 12:14:20.848308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.848234 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 12:14:20.955634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:20.955608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:20.955715 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.955690 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:20.955767 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:20.955731 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:21.955718927 +0000 UTC m=+2.954261262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:21.056635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.056614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:21.056733 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.056651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:21.056733 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056723 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:21.056841 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056760 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:22.056749296 +0000 UTC m=+3.055291625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:21.056841 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056778 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:21.056841 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056799 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:21.056841 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056812 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:21.056984 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:21.056867 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:22.056851054 +0000 UTC m=+3.055393403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:21.106769 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.106726 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 12:14:21.398717 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.398629 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:21 +0000 UTC" deadline="2028-05-28 17:10:33.663575186 +0000 UTC" Sep 11 12:14:21.398717 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.398670 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15004h56m12.264909591s" Sep 11 12:14:21.441790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.441733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6h22t" event={"ID":"a073adff-9215-4929-8b7a-4ba3b3522f50","Type":"ContainerStarted","Data":"31a660dab39ac3100e7df715eda37313cf608236d82feeaeb72b90b5e8977faf"} Sep 11 12:14:21.444480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.444424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerStarted","Data":"e971691aba4450a0e83442b83d80a9d78167f7733c1c7f9be239b569b12f27ac"} Sep 11 12:14:21.445912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.445860 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86dcp" event={"ID":"a236ce12-2ee8-42a3-9dc6-82c584e91f24","Type":"ContainerStarted","Data":"aad8c480749eae2d23d9fb18eb74d341c80728e74f65388019316a3584846526"} Sep 11 12:14:21.452830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.452781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5wrfm" event={"ID":"b34698e3-de0b-4d89-8c31-901d9efea428","Type":"ContainerStarted","Data":"ac7d5ddb5ae205b2a1ad6d86d1e0b31808490051659ebd407c3a276907c01ecb"} Sep 11 12:14:21.458866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.458837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" event={"ID":"c58c631d-96c5-4da1-b807-41709d259e3b","Type":"ContainerStarted","Data":"4eee0fade97a0c251070b5970ecd8d774fe52b18015f358c9fd628316e4a8906"} Sep 11 12:14:21.461091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.461063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w4tw4" event={"ID":"0ba65492-6ff4-489f-b229-3fe6896728aa","Type":"ContainerStarted","Data":"50e68e7d68426e22be1b8cbf995eeca02b73c20e6a39850e1da7dd914455b9bf"} Sep 11 12:14:21.464000 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.463971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"de84f38308edebd4813e8bd07878fae2366162c4ad7a30bb93ac288faa1cad5f"} Sep 11 12:14:21.467712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.467686 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mznf6" event={"ID":"02643625-d215-4959-a4b6-2cd85e9ad9a2","Type":"ContainerStarted","Data":"ebb12cfcde0d6f5364f3d0ffc4815b07139fc98436152793a127427a44f5791a"} Sep 11 12:14:21.469980 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.469955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" event={"ID":"66903a53d0aed29fd4572d053e31c420","Type":"ContainerStarted","Data":"bf3b11b9d1cd4a39203f8aeee65c549eb7dd6b670031a328cf47dbb220579fb6"} Sep 11 12:14:21.472377 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.472350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"f7d3bac48cef4011d55130ca1dfcfcce","Type":"ContainerStarted","Data":"f397d133c70c54307f62bd4f012d8155f68efde76b1534f0ddf56eadb9270f00"} Sep 11 12:14:21.474871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:21.474845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbtsc" event={"ID":"819dccc0-085c-40a5-b00c-80b876a761db","Type":"ContainerStarted","Data":"6c57326bafc26a0f2328775d64cd69fb37998f6e998ed7e1908f6bde45b0ad25"} Sep 11 12:14:23.474597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.474546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:23.474801 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.474724 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:23.474801 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.474788 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:25.474770522 +0000 UTC m=+4.962975557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:23.575290 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.575249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:23.575472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.575334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:23.575535 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.575473 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:23.575586 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.575538 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:25.575518514 +0000 UTC m=+5.063723569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:23.575997 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.575977 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:23.576089 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.576002 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:23.576089 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.576015 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:23.576089 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.576059 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:25.576045255 +0000 UTC m=+5.064250295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:23.909484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.909371 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 12:09:21 +0000 UTC" deadline="2028-03-11 05:18:18.755817273 +0000 UTC" Sep 11 12:14:23.909484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.909434 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13121h3m54.846387028s" Sep 11 12:14:23.946866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.946829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:23.947057 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.946962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:23.947427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.947405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:23.947518 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.947502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:23.947601 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:23.947588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:23.947707 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:23.947691 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:25.503704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.503667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:25.504283 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.503820 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:25.504283 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.503886 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:29.503864953 +0000 UTC m=+8.992070008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:25.604667 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.604558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:25.604667 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.604660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:25.604880 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.604809 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:25.604880 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.604829 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:25.604880 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.604841 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:25.605029 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.604895 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:29.60487765 +0000 UTC m=+9.093082685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:25.605895 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.605272 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:25.605895 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.605325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:29.605310385 +0000 UTC m=+9.093515424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:25.946910 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.946837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:25.947076 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.946982 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:25.947282 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.947263 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:25.947348 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.947338 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:25.947398 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:25.947369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:25.947444 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:25.947411 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:27.946772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:27.946906 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:27.947264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:27.947357 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:27.947417 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:27.947589 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:27.947484 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:29.536017 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.535980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:29.536441 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.536118 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:29.536441 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.536185 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:37.536168661 +0000 UTC m=+17.024373701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:29.636770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.636731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:29.636933 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.636811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:29.636983 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.636959 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:29.636983 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.636975 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:29.637112 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.636984 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:29.637112 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.637033 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:37.637018688 +0000 UTC m=+17.125223723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:29.637112 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.637088 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:29.637226 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.637119 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:37.637110705 +0000 UTC m=+17.125315740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.946685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.946717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:29.946685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.946813 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.946905 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:29.947156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:29.946980 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:31.946427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:31.946453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:31.946433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:31.946594 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:31.946654 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:31.946946 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:31.946707 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:33.946540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:33.946503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:33.946994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:33.946503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:33.946994 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:33.946651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:33.946994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:33.946503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:33.946994 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:33.946697 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:33.946994 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:33.946759 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:35.947328 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:35.947297 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:35.947883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:35.947311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:35.947883 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:35.947419 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:35.947883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:35.947429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:35.947883 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:35.947500 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:35.947883 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:35.947551 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:37.600429 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.600391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:37.600864 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.600530 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:37.600864 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.600599 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:53.600578331 +0000 UTC m=+33.088783371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:37.701177 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.701128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.701200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701292 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701319 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701337 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701351 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:37.701359 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701364 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:14:53.701345194 +0000 UTC m=+33.189550242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:37.701694 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.701394 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:53.701377934 +0000 UTC m=+33.189582974 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:37.947457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.947372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:37.947457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.947422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:37.947768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:37.947495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:37.947768 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.947505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:37.947768 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.947629 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:37.947768 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:37.947681 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:39.946626 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:39.946579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:39.947055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:39.946637 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:39.947055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:39.946717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:39.947055 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:39.946727 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:39.947055 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:39.946840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:39.947055 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:39.946935 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:41.947165 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:41.947134 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:41.947165 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:41.947150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:41.947467 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:41.947233 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:41.947467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:41.947269 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:41.947467 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:41.947342 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:41.947467 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:41.947374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:43.035710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.035316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" event={"ID":"66903a53d0aed29fd4572d053e31c420","Type":"ContainerStarted","Data":"1fa6fb6c88593c2ed801ba17198ea6d27a3db6e29a35ce2ddb6bed457f2fc49b"} Sep 11 12:14:43.037010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.036987 2575 generic.go:356] "Generic (PLEG): container finished" podID="f7d3bac48cef4011d55130ca1dfcfcce" containerID="17ca9db7182022c9c881a81e9e032bddd4ea84b82caab4fc56116478f44b7e2f" exitCode=0 Sep 11 12:14:43.037132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.037073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"f7d3bac48cef4011d55130ca1dfcfcce","Type":"ContainerDied","Data":"17ca9db7182022c9c881a81e9e032bddd4ea84b82caab4fc56116478f44b7e2f"} Sep 11 12:14:43.038663 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.038639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbtsc" event={"ID":"819dccc0-085c-40a5-b00c-80b876a761db","Type":"ContainerStarted","Data":"a7f05a0dfeb0fd898404664b453440658875f194cbed009f7b9e2e60573ee51e"} Sep 11 12:14:43.040151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.040124 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="e7523b2f9b25f484ed6eaecbf1f4f354cc3610e99058bacc54a286646dfb9824" exitCode=0 Sep 11 12:14:43.040347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.040218 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"e7523b2f9b25f484ed6eaecbf1f4f354cc3610e99058bacc54a286646dfb9824"} Sep 11 12:14:43.041787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.041769 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86dcp" event={"ID":"a236ce12-2ee8-42a3-9dc6-82c584e91f24","Type":"ContainerStarted","Data":"4a65696ecc49f5a7c7cb9cc12ec4e1d70ee309e75dcc8bb6e675cd8d19eaf8ff"} Sep 11 12:14:43.043342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.043323 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5wrfm" event={"ID":"b34698e3-de0b-4d89-8c31-901d9efea428","Type":"ContainerStarted","Data":"1bf1c3c46619a090942e115dfcd0f704c3033638648a35c7a3a8565afa2800e5"} Sep 11 12:14:43.044818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.044795 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" event={"ID":"c58c631d-96c5-4da1-b807-41709d259e3b","Type":"ContainerStarted","Data":"fb9cb345e788675cade03f13d9e381bc7187a823abe752251ebdae0a49ee7b9a"} Sep 11 12:14:43.046269 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.046251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w4tw4" event={"ID":"0ba65492-6ff4-489f-b229-3fe6896728aa","Type":"ContainerStarted","Data":"1de23d518336a1ebfacbbd58cb766a801d29c574cf34bd39a3e908776b7f8927"} Sep 11 12:14:43.049222 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"fb1189633338e6c6be5a9cfd191ab23d17da5d7bcccf028882a05403ce714bff"} Sep 11 12:14:43.049311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"a46f1e5fdd6f2268ab18c5e60dd0b9d117d58032d9ba08335c15fd0e60840de2"} Sep 11 12:14:43.049311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049281 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"c0eda484eb9f6408a067f54f2eeb168fed0099f5542178a7e16e4ff06562562d"} Sep 11 12:14:43.049311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"c228d70402a5e828d2cc442f666152446807ea39e60e339b42f86ae2ac08235f"} Sep 11 12:14:43.049455 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"ee52474f0e6228eadb1b632ca32fc03e9d020d62d0e369432ee21317fbb4a709"} Sep 11 12:14:43.049455 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.049364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"3c38928763b0b70a0884ed518f0b47daf6ba957dc4723e38332fb45b7e4793a1"} Sep 11 12:14:43.050506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.050480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mznf6" event={"ID":"02643625-d215-4959-a4b6-2cd85e9ad9a2","Type":"ContainerStarted","Data":"1ce4d2382ae9bae7cd903962f315ef77370ab450a0dde3c3ef6bc9d5d7c1dbe9"} Sep 11 12:14:43.057198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.057161 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-239.ec2.internal" podStartSLOduration=22.057148094 podStartE2EDuration="22.057148094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.056178403 +0000 UTC m=+22.544383459" watchObservedRunningTime="2026-09-11 12:14:43.057148094 +0000 UTC m=+22.545353227" Sep 11 12:14:43.086549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.086487 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mznf6" podStartSLOduration=2.352865648 podStartE2EDuration="22.086468403s" totalImagesPullingTime="19.733602755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.085718919 +0000 UTC m=+22.573923977" watchObservedRunningTime="2026-09-11 12:14:43.086468403 +0000 UTC m=+22.574673462" Sep 11 12:14:43.117249 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.116760 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w4tw4" podStartSLOduration=2.478592348 podStartE2EDuration="22.116740067s" totalImagesPullingTime="19.638147719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.116409602 +0000 UTC m=+22.604614659" watchObservedRunningTime="2026-09-11 12:14:43.116740067 +0000 UTC m=+22.604945125" Sep 11 12:14:43.179749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.179707 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbtsc" podStartSLOduration=2.637767139 podStartE2EDuration="22.179691411s" totalImagesPullingTime="19.541924272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.179552066 +0000 UTC m=+22.667757122" watchObservedRunningTime="2026-09-11 12:14:43.179691411 +0000 UTC m=+22.667896468" Sep 11 12:14:43.180022 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.180002 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5wrfm" podStartSLOduration=2.485348458 podStartE2EDuration="22.179996628s" totalImagesPullingTime="19.69464817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.146634077 +0000 UTC m=+22.634839133" watchObservedRunningTime="2026-09-11 12:14:43.179996628 +0000 UTC m=+22.668201684" Sep 11 12:14:43.230260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.228227 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-86dcp" podStartSLOduration=2.547390058 podStartE2EDuration="22.228208743s" totalImagesPullingTime="19.680818685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:43.226973858 +0000 UTC m=+22.715178914" watchObservedRunningTime="2026-09-11 12:14:43.228208743 +0000 UTC m=+22.716413801" Sep 11 12:14:43.946764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.946734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:43.947043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.946851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:43.947043 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:43.946852 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:43.947043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:43.946850 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:43.947043 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:43.946927 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:43.947043 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:43.947026 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:44.055229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.055186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" event={"ID":"f7d3bac48cef4011d55130ca1dfcfcce","Type":"ContainerStarted","Data":"e75396f28f8120698212ad31e7471656abd003a46af346b93fc7f38804181b12"} Sep 11 12:14:44.058113 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.057900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6h22t" event={"ID":"a073adff-9215-4929-8b7a-4ba3b3522f50","Type":"ContainerStarted","Data":"ff27bdecc3acf8d55a95e26a1604300316539fc00b2a36984072623790749e9f"} Sep 11 12:14:44.073103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.072911 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-239.ec2.internal" podStartSLOduration=23.072894013 podStartE2EDuration="23.072894013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:44.072472468 +0000 UTC m=+23.560677526" watchObservedRunningTime="2026-09-11 12:14:44.072894013 +0000 UTC m=+23.561099105" Sep 11 12:14:44.090979 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.090931 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6h22t" podStartSLOduration=3.434716864 podStartE2EDuration="23.090912909s" totalImagesPullingTime="19.656196045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:44.090834698 +0000 UTC m=+23.579039756" watchObservedRunningTime="2026-09-11 12:14:44.090912909 +0000 UTC m=+23.579117965" Sep 11 12:14:44.136465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.136439 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 12:14:44.781190 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.781161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:44.781742 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.781721 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:44.924862 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.924748 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T12:14:44.136456811Z","UUID":"86c694dd-9bba-481f-89eb-acab32c0a6ef","Handler":null,"Name":"","Endpoint":""} Sep 11 12:14:44.926382 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.926364 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 12:14:44.926494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:44.926398 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 12:14:45.062959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.062794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" event={"ID":"c58c631d-96c5-4da1-b807-41709d259e3b","Type":"ContainerStarted","Data":"8b572d20c27fb69466af4fd6c01dfb593bfc9933bdec5ad0ef2e6e974dabe45e"} Sep 11 12:14:45.068040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.068013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"58c068d1a4e9171f965ef9760e78ae3327d4bef5dd7599a2ebeb1930d11cdc8f"} Sep 11 12:14:45.068170 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.068089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:45.069246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.069225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbtsc" Sep 11 12:14:45.947151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.947127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:45.947316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.947223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:45.947316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:45.947228 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:45.947430 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:45.947319 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:45.947430 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:45.947390 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:45.947430 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:45.947222 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:46.072388 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:46.072355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" event={"ID":"c58c631d-96c5-4da1-b807-41709d259e3b","Type":"ContainerStarted","Data":"c58d7723bbfc4226e3cda14375df2b334a4b0c2da54d740d55f93a0a4a565890"} Sep 11 12:14:46.095979 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:46.095938 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9d4v" podStartSLOduration=2.383094999 podStartE2EDuration="25.095926417s" totalImagesPullingTime="22.712831418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:46.094422527 +0000 UTC m=+25.582627584" watchObservedRunningTime="2026-09-11 12:14:46.095926417 +0000 UTC m=+25.584131474" Sep 11 12:14:47.947221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:47.947190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:47.947729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:47.947196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:47.947729 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:47.947334 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:47.947729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:47.947196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:47.947729 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:47.947405 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:47.947729 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:47.947444 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:49.084045 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.083814 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="a8d9e27f361ba9da812be9eb7596630d71edf0b0e99fdeeea229dc3b950c1e5f" exitCode=0 Sep 11 12:14:49.084454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.083903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"a8d9e27f361ba9da812be9eb7596630d71edf0b0e99fdeeea229dc3b950c1e5f"} Sep 11 12:14:49.088765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.088728 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" event={"ID":"47381769-db6f-4f64-be82-c0561af5ec74","Type":"ContainerStarted","Data":"4cd48550c5e8583cf49dce0cd00e598a08e450b72c6d5079204adfe221edbb9d"} Sep 11 12:14:49.089179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.089015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:49.089179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.089096 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:49.089179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.089145 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:49.106064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.106040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:49.106252 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.106234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:14:49.151460 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.151407 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" podStartSLOduration=8.432429996 podStartE2EDuration="28.151395159s" totalImagesPullingTime="19.718965163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:14:49.148993847 +0000 UTC m=+28.637198920" watchObservedRunningTime="2026-09-11 12:14:49.151395159 +0000 UTC m=+28.639600215" Sep 11 12:14:49.946653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.946596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:49.946926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.946624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:49.946926 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:49.946763 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:49.946926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:49.946631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:49.946926 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:49.946848 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:49.946926 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:49.946889 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:50.175236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.175211 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glbfl"] Sep 11 12:14:50.175856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.175295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:50.175856 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:50.175379 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:50.177964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.177943 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6vhr"] Sep 11 12:14:50.178084 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.178017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:50.178142 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:50.178098 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:50.181237 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.181218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9mtmd"] Sep 11 12:14:50.181316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:50.181285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:50.181366 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:50.181349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:51.947122 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:51.946973 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:51.947474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:51.947032 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:51.947474 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:51.947215 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:51.947474 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:51.947303 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:51.947474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:51.947040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:51.947474 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:51.947457 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:52.096433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:52.096405 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="c41463dad41b19a6807e3bc41f8fdf282d8b713b353296c7fd8c0ce11bbfd382" exitCode=0 Sep 11 12:14:52.096558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:52.096466 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"c41463dad41b19a6807e3bc41f8fdf282d8b713b353296c7fd8c0ce11bbfd382"} Sep 11 12:14:53.618228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.618201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:53.618514 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.618307 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:53.618514 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.618352 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs podName:34c7c065-ce6f-4bc4-a44b-5e00d23b880b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:25.618339658 +0000 UTC m=+65.106544694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs") pod "network-metrics-daemon-9mtmd" (UID: "34c7c065-ce6f-4bc4-a44b-5e00d23b880b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 12:14:53.719467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.719412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:53.719467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.719462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:53.719624 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719554 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:53.719624 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719562 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 12:14:53.719624 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719577 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 12:14:53.719624 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719585 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jhrkx for pod openshift-network-diagnostics/network-check-target-g6vhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:53.719624 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719626 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret podName:b2b3ef44-c676-4dbe-bd64-17e7a276afee nodeName:}" failed. No retries permitted until 2026-09-11 12:15:25.719595351 +0000 UTC m=+65.207800385 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret") pod "global-pull-secret-syncer-glbfl" (UID: "b2b3ef44-c676-4dbe-bd64-17e7a276afee") : object "kube-system"/"original-pull-secret" not registered Sep 11 12:14:53.719807 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.719644 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx podName:5a7a660c-88c9-4eea-bc99-fc47ba8675f1 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:25.71963192 +0000 UTC m=+65.207836958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhrkx" (UniqueName: "kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx") pod "network-check-target-g6vhr" (UID: "5a7a660c-88c9-4eea-bc99-fc47ba8675f1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 12:14:53.946677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.946647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:53.946815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.946674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:53.946815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:53.946647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:53.946815 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.946741 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g6vhr" podUID="5a7a660c-88c9-4eea-bc99-fc47ba8675f1" Sep 11 12:14:53.946915 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.946834 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9mtmd" podUID="34c7c065-ce6f-4bc4-a44b-5e00d23b880b" Sep 11 12:14:53.946915 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:53.946907 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-glbfl" podUID="b2b3ef44-c676-4dbe-bd64-17e7a276afee" Sep 11 12:14:54.102596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:54.102522 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="a49d4d8926267ea7703bf0e3d6996629d50f1bdd984891bc8d7a8ea0da89a095" exitCode=0 Sep 11 12:14:54.102708 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:54.102595 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"a49d4d8926267ea7703bf0e3d6996629d50f1bdd984891bc8d7a8ea0da89a095"} Sep 11 12:14:55.859040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.859012 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-239.ec2.internal" event="NodeReady" Sep 11 12:14:55.859430 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.859150 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 12:14:55.900504 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.900480 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:14:55.921558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.921537 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p5kc7"] Sep 11 12:14:55.921718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.921697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:55.924205 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.924183 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 12:14:55.924328 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.924259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 12:14:55.924528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.924509 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 12:14:55.924654 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.924550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bmqgw\"" Sep 11 12:14:55.939504 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.939478 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 12:14:55.939974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.939958 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:14:55.940082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.939992 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r9tvs"] Sep 11 12:14:55.940140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.940081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:55.942884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.942867 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\"" Sep 11 12:14:55.943187 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.943170 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 12:14:55.943261 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.943206 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 12:14:55.959655 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959633 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p5kc7"] Sep 11 12:14:55.959751 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9tvs"] Sep 11 12:14:55.959751 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:55.959858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:14:55.959858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:14:55.959858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.959784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.965973 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ld655\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.965998 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.966012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.966124 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.966357 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.966378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 12:14:55.967359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.966435 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 12:14:55.968040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.968020 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 12:14:55.968250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.968222 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\"" Sep 11 12:14:55.968445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:55.968429 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\"" Sep 11 12:14:56.036082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.036193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036122 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftzvt\" (UniqueName: \"kubernetes.io/projected/17d41008-bde7-4c58-970c-840b80e7165b-kube-api-access-ftzvt\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.036301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17d41008-bde7-4c58-970c-840b80e7165b-config-volume\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.036301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d41008-bde7-4c58-970c-840b80e7165b-tmp-dir\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.036546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r64h\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.036546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.036393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137014 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.137149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftzvt\" (UniqueName: \"kubernetes.io/projected/17d41008-bde7-4c58-970c-840b80e7165b-kube-api-access-ftzvt\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17d41008-bde7-4c58-970c-840b80e7165b-config-volume\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d41008-bde7-4c58-970c-840b80e7165b-tmp-dir\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.137318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r64h\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137630 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137630 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecdce906-182f-43e2-b55a-6269548d60b6-cert\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.137630 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137630 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlw9\" (UniqueName: \"kubernetes.io/projected/ecdce906-182f-43e2-b55a-6269548d60b6-kube-api-access-xhlw9\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.137675 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d41008-bde7-4c58-970c-840b80e7165b-tmp-dir\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.137180 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.137696 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.137710 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.137737 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:56.63772235 +0000 UTC m=+36.125927384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:14:56.137855 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.137758 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:56.637746883 +0000 UTC m=+36.125951918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:14:56.138208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.138108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.138349 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.138327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17d41008-bde7-4c58-970c-840b80e7165b-config-volume\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.138421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.138353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.142344 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.142321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.142447 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.142431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.149336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.149313 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.149336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.149327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r64h\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.159426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.159409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftzvt\" (UniqueName: \"kubernetes.io/projected/17d41008-bde7-4c58-970c-840b80e7165b-kube-api-access-ftzvt\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.238170 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.238121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecdce906-182f-43e2-b55a-6269548d60b6-cert\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.238170 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.238161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlw9\" (UniqueName: \"kubernetes.io/projected/ecdce906-182f-43e2-b55a-6269548d60b6-kube-api-access-xhlw9\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.240539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.240520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecdce906-182f-43e2-b55a-6269548d60b6-cert\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.252881 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.252860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlw9\" (UniqueName: \"kubernetes.io/projected/ecdce906-182f-43e2-b55a-6269548d60b6-kube-api-access-xhlw9\") pod \"ingress-canary-r9tvs\" (UID: \"ecdce906-182f-43e2-b55a-6269548d60b6\") " pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.270683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.270660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r9tvs" Sep 11 12:14:56.589480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.589254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r9tvs"] Sep 11 12:14:56.593708 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:56.593684 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecdce906_182f_43e2_b55a_6269548d60b6.slice/crio-f9a59538f6572096091316bb85725ba0313a916b1f6c47946975eaaf5f1c9b1f WatchSource:0}: Error finding container f9a59538f6572096091316bb85725ba0313a916b1f6c47946975eaaf5f1c9b1f: Status 404 returned error can't find the container with id f9a59538f6572096091316bb85725ba0313a916b1f6c47946975eaaf5f1c9b1f Sep 11 12:14:56.641689 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.641664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:56.641767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.641723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:56.641818 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.641806 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:14:56.641818 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.641806 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:14:56.641892 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.641816 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:14:56.641892 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.641863 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.641848379 +0000 UTC m=+37.130053414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:14:56.641892 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:56.641876 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.641870725 +0000 UTC m=+37.130075759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:14:56.712184 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.712157 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v"] Sep 11 12:14:56.744205 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.744183 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v"] Sep 11 12:14:56.744319 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.744222 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc"] Sep 11 12:14:56.744319 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.744302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" Sep 11 12:14:56.746938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.746914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.747079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.746962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.747425 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.747405 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-r6j7v\"" Sep 11 12:14:56.758088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.758071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc"] Sep 11 12:14:56.758178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.758151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:56.760872 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.760850 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 12:14:56.760971 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.760875 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.760971 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.760927 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w5whl\"" Sep 11 12:14:56.761221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.761201 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.836941 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.836918 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-s9tjz"] Sep 11 12:14:56.843343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.843299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/e8df42ea-beeb-4e7b-92a0-1d23cd64181c-kube-api-access-7t6pj\") pod \"volume-data-source-validator-86695b7cc7-pnj8v\" (UID: \"e8df42ea-beeb-4e7b-92a0-1d23cd64181c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" Sep 11 12:14:56.854073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.854056 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nqz29"] Sep 11 12:14:56.854230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.854212 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:56.868073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.868055 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64"] Sep 11 12:14:56.868355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.868241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" Sep 11 12:14:56.874944 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.874915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9xx4t\"" Sep 11 12:14:56.877001 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.876981 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 12:14:56.877192 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.877176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.878069 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.877894 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.878392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.878374 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 12:14:56.883483 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.883465 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dn6s8\"" Sep 11 12:14:56.888230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.888211 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68f4bb8d9b-qnbg7"] Sep 11 12:14:56.888378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.888361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:56.889229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.889187 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 12:14:56.891743 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.891712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.891966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.891947 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 12:14:56.892266 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.892249 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.892958 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.892932 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cd8fd\"" Sep 11 12:14:56.893177 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.893164 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 12:14:56.910441 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.910421 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z"] Sep 11 12:14:56.910586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.910567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:56.913214 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.913196 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.913760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.913741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 12:14:56.913842 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.913787 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ljcfs\"" Sep 11 12:14:56.914022 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.914004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 12:14:56.914258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.914245 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 12:14:56.914441 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.914424 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 12:14:56.914931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.914913 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.920475 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920451 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nqz29"] Sep 11 12:14:56.920549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s9tjz"] Sep 11 12:14:56.920549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920504 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z"] Sep 11 12:14:56.920549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920524 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64"] Sep 11 12:14:56.920549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f4bb8d9b-qnbg7"] Sep 11 12:14:56.920732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.920556 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:56.926568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.926191 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 12:14:56.926568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.926248 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.926568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.926316 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rps94\"" Sep 11 12:14:56.926568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.926324 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.926568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.926372 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 12:14:56.935274 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.935253 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn"] Sep 11 12:14:56.944657 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.944634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzsb\" (UniqueName: \"kubernetes.io/projected/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-kube-api-access-pxzsb\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:56.944767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.944750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/e8df42ea-beeb-4e7b-92a0-1d23cd64181c-kube-api-access-7t6pj\") pod \"volume-data-source-validator-86695b7cc7-pnj8v\" (UID: \"e8df42ea-beeb-4e7b-92a0-1d23cd64181c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" Sep 11 12:14:56.944833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.944782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:56.955355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.954699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:56.958405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.958380 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 12:14:56.958498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.958409 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vs6pw\"" Sep 11 12:14:56.958774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.958755 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dbnv2"] Sep 11 12:14:56.959256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.958996 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.959256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.959091 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 12:14:56.959256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.959165 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.963336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.963319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6pj\" (UniqueName: \"kubernetes.io/projected/e8df42ea-beeb-4e7b-92a0-1d23cd64181c-kube-api-access-7t6pj\") pod \"volume-data-source-validator-86695b7cc7-pnj8v\" (UID: \"e8df42ea-beeb-4e7b-92a0-1d23cd64181c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" Sep 11 12:14:56.976669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.976650 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn"] Sep 11 12:14:56.976752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.976681 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dbnv2"] Sep 11 12:14:56.976796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.976778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:56.979498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.979477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 12:14:56.979652 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.979635 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 12:14:56.979764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.979750 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:14:56.979986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.979971 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 12:14:56.980788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.980769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tv95s\"" Sep 11 12:14:56.986817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:56.986795 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 12:14:57.046035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0874441a-169d-48eb-9424-330a62809713-serving-cert\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.046162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046057 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:57.046162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn76m\" (UniqueName: \"kubernetes.io/projected/0874441a-169d-48eb-9424-330a62809713-kube-api-access-pn76m\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.046162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7tb\" (UniqueName: \"kubernetes.io/projected/d26cd9a3-d26f-4edd-b61e-face2237376e-kube-api-access-fl7tb\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.046305 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.046214 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:14:57.046305 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzsb\" (UniqueName: \"kubernetes.io/projected/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-kube-api-access-pxzsb\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:57.046305 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.046305 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.046281 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.546260494 +0000 UTC m=+37.034465546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:14:57.046485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b889fdfc-8f57-4240-abd7-1a1ed43366aa-serving-cert\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.046485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d26cd9a3-d26f-4edd-b61e-face2237376e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.046485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-stats-auth\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.046485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b889fdfc-8f57-4240-abd7-1a1ed43366aa-config\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.046485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-tmp\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqxw\" (UniqueName: \"kubernetes.io/projected/b889fdfc-8f57-4240-abd7-1a1ed43366aa-kube-api-access-zsqxw\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-snapshots\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85677\" (UniqueName: \"kubernetes.io/projected/29c80efb-0d0b-44ba-b8ec-95d061729afa-kube-api-access-85677\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.046748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.047118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-service-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.047118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdhz\" (UniqueName: \"kubernetes.io/projected/e75bc421-07a1-49a9-b0e3-83fc5908fbae-kube-api-access-pwdhz\") pod \"network-check-source-6f8b768455-nqz29\" (UID: \"e75bc421-07a1-49a9-b0e3-83fc5908fbae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" Sep 11 12:14:57.047118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.046880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-default-certificate\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.055210 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.055184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" Sep 11 12:14:57.055581 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.055559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzsb\" (UniqueName: \"kubernetes.io/projected/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-kube-api-access-pxzsb\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:57.114055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.113921 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9tvs" event={"ID":"ecdce906-182f-43e2-b55a-6269548d60b6","Type":"ContainerStarted","Data":"f9a59538f6572096091316bb85725ba0313a916b1f6c47946975eaaf5f1c9b1f"} Sep 11 12:14:57.147704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn76m\" (UniqueName: \"kubernetes.io/projected/0874441a-169d-48eb-9424-330a62809713-kube-api-access-pn76m\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.147803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147709 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7tb\" (UniqueName: \"kubernetes.io/projected/d26cd9a3-d26f-4edd-b61e-face2237376e-kube-api-access-fl7tb\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.147803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrcm\" (UniqueName: \"kubernetes.io/projected/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-kube-api-access-khrcm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.147803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.147803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b889fdfc-8f57-4240-abd7-1a1ed43366aa-serving-cert\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.147926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d26cd9a3-d26f-4edd-b61e-face2237376e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.147926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-stats-auth\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.147926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b889fdfc-8f57-4240-abd7-1a1ed43366aa-config\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.147926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-tmp\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.148039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147950 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-config\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.148039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.147973 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wj6\" (UniqueName: \"kubernetes.io/projected/8a591965-0202-4194-b784-8aa97294024c-kube-api-access-q7wj6\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.148039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a591965-0202-4194-b784-8aa97294024c-serving-cert\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.148039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.148154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.148154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqxw\" (UniqueName: \"kubernetes.io/projected/b889fdfc-8f57-4240-abd7-1a1ed43366aa-kube-api-access-zsqxw\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.148154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.148154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-snapshots\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-trusted-ca\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85677\" (UniqueName: \"kubernetes.io/projected/29c80efb-0d0b-44ba-b8ec-95d061729afa-kube-api-access-85677\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-tmp\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.148316 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.648293947 +0000 UTC m=+37.136499006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:14:57.148364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-service-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.148739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdhz\" (UniqueName: \"kubernetes.io/projected/e75bc421-07a1-49a9-b0e3-83fc5908fbae-kube-api-access-pwdhz\") pod \"network-check-source-6f8b768455-nqz29\" (UID: \"e75bc421-07a1-49a9-b0e3-83fc5908fbae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" Sep 11 12:14:57.148739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-default-certificate\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.148739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.148489 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0874441a-169d-48eb-9424-330a62809713-serving-cert\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.149047 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.149022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d26cd9a3-d26f-4edd-b61e-face2237376e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.149100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.149048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.149137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.149124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0874441a-169d-48eb-9424-330a62809713-service-ca-bundle\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.149215 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.149202 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:14:57.149265 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.149255 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.649241079 +0000 UTC m=+37.137446115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:14:57.149509 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.149490 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:57.149590 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.149557 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:14:57.649541297 +0000 UTC m=+37.137746333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:57.149847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.149819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b889fdfc-8f57-4240-abd7-1a1ed43366aa-config\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.150215 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.150190 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0874441a-169d-48eb-9424-330a62809713-snapshots\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.150787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.150733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b889fdfc-8f57-4240-abd7-1a1ed43366aa-serving-cert\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.151738 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.151719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-stats-auth\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.152065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.152048 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-default-certificate\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.152976 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.152954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0874441a-169d-48eb-9424-330a62809713-serving-cert\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.167724 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.167684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7tb\" (UniqueName: \"kubernetes.io/projected/d26cd9a3-d26f-4edd-b61e-face2237376e-kube-api-access-fl7tb\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.169027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.169007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn76m\" (UniqueName: \"kubernetes.io/projected/0874441a-169d-48eb-9424-330a62809713-kube-api-access-pn76m\") pod \"insights-operator-5844b856db-s9tjz\" (UID: \"0874441a-169d-48eb-9424-330a62809713\") " pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.169859 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.169811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85677\" (UniqueName: \"kubernetes.io/projected/29c80efb-0d0b-44ba-b8ec-95d061729afa-kube-api-access-85677\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.170630 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.170593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqxw\" (UniqueName: \"kubernetes.io/projected/b889fdfc-8f57-4240-abd7-1a1ed43366aa-kube-api-access-zsqxw\") pod \"service-ca-operator-cf7c776b7-qrt64\" (UID: \"b889fdfc-8f57-4240-abd7-1a1ed43366aa\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.171744 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.171678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdhz\" (UniqueName: \"kubernetes.io/projected/e75bc421-07a1-49a9-b0e3-83fc5908fbae-kube-api-access-pwdhz\") pod \"network-check-source-6f8b768455-nqz29\" (UID: \"e75bc421-07a1-49a9-b0e3-83fc5908fbae\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" Sep 11 12:14:57.181767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.181748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" Sep 11 12:14:57.199223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.199206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.249901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrcm\" (UniqueName: \"kubernetes.io/projected/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-kube-api-access-khrcm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-config\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wj6\" (UniqueName: \"kubernetes.io/projected/8a591965-0202-4194-b784-8aa97294024c-kube-api-access-q7wj6\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a591965-0202-4194-b784-8aa97294024c-serving-cert\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.250240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-trusted-ca\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.251192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-trusted-ca\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.252592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.252239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a591965-0202-4194-b784-8aa97294024c-config\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.253451 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.253046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.256785 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.254496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.258041 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.258019 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a591965-0202-4194-b784-8aa97294024c-serving-cert\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.262437 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.262418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrcm\" (UniqueName: \"kubernetes.io/projected/8da8d03e-56f4-4451-b35a-4e4a4dfe6f72-kube-api-access-khrcm\") pod \"kube-storage-version-migrator-operator-6c8f84f897-tjpvn\" (UID: \"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.262546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.262503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wj6\" (UniqueName: \"kubernetes.io/projected/8a591965-0202-4194-b784-8aa97294024c-kube-api-access-q7wj6\") pod \"console-operator-696884fb9c-dbnv2\" (UID: \"8a591965-0202-4194-b784-8aa97294024c\") " pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.266022 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.265989 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" Sep 11 12:14:57.291075 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.291053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:14:57.354669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.354629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v"] Sep 11 12:14:57.359944 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:57.359885 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8df42ea_beeb_4e7b_92a0_1d23cd64181c.slice/crio-7b220d2fb6c900fb711a5321da10668d7092bf8a2172048d5c6842ce2c29a72b WatchSource:0}: Error finding container 7b220d2fb6c900fb711a5321da10668d7092bf8a2172048d5c6842ce2c29a72b: Status 404 returned error can't find the container with id 7b220d2fb6c900fb711a5321da10668d7092bf8a2172048d5c6842ce2c29a72b Sep 11 12:14:57.463848 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.463815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s9tjz" Sep 11 12:14:57.476126 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.476100 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-nqz29"] Sep 11 12:14:57.477902 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:57.477870 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75bc421_07a1_49a9_b0e3_83fc5908fbae.slice/crio-29fe0ae8957c9b572f3657af6194ef1d079c38c1d403c211da07b9810c3a9c9c WatchSource:0}: Error finding container 29fe0ae8957c9b572f3657af6194ef1d079c38c1d403c211da07b9810c3a9c9c: Status 404 returned error can't find the container with id 29fe0ae8957c9b572f3657af6194ef1d079c38c1d403c211da07b9810c3a9c9c Sep 11 12:14:57.528789 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.528758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64"] Sep 11 12:14:57.534456 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:57.534429 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb889fdfc_8f57_4240_abd7_1a1ed43366aa.slice/crio-b769f63cb3681ceb5ff9c126529cf414ebbca15dcf4e9ed7d42d2a93e00e3cab WatchSource:0}: Error finding container b769f63cb3681ceb5ff9c126529cf414ebbca15dcf4e9ed7d42d2a93e00e3cab: Status 404 returned error can't find the container with id b769f63cb3681ceb5ff9c126529cf414ebbca15dcf4e9ed7d42d2a93e00e3cab Sep 11 12:14:57.554363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.553996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:57.554363 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.554228 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:14:57.554363 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.554289 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.554271586 +0000 UTC m=+38.042476637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:14:57.579528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.579500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn"] Sep 11 12:14:57.586604 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:57.586574 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da8d03e_56f4_4451_b35a_4e4a4dfe6f72.slice/crio-142d0765228f5a37091a1912010d89e2235c8fca81fd552e58151d29aad7287e WatchSource:0}: Error finding container 142d0765228f5a37091a1912010d89e2235c8fca81fd552e58151d29aad7287e: Status 404 returned error can't find the container with id 142d0765228f5a37091a1912010d89e2235c8fca81fd552e58151d29aad7287e Sep 11 12:14:57.628190 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.628130 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dbnv2"] Sep 11 12:14:57.630580 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:14:57.630546 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a591965_0202_4194_b784_8aa97294024c.slice/crio-b96997fc6c56b9bf3be8cd339382297e59a3f613bd3abfc426bdd904f4d8a4bd WatchSource:0}: Error finding container b96997fc6c56b9bf3be8cd339382297e59a3f613bd3abfc426bdd904f4d8a4bd: Status 404 returned error can't find the container with id b96997fc6c56b9bf3be8cd339382297e59a3f613bd3abfc426bdd904f4d8a4bd Sep 11 12:14:57.655210 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.655190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:57.655288 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.655232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:57.655343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.655305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.655387 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655336 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:14:57.655387 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655354 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:14:57.655454 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655408 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:14:59.655387853 +0000 UTC m=+39.143592888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:14:57.655454 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655417 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:14:57.655454 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655440 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.655424227 +0000 UTC m=+38.143629287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:14:57.655585 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655469 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:14:59.655455565 +0000 UTC m=+39.143660604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:14:57.655585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.655488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:57.655585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.655515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:57.655764 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655591 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:14:57.655764 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655640 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:57.655764 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655661 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.655647505 +0000 UTC m=+38.143852540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:14:57.655764 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:57.655678 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:14:58.655668431 +0000 UTC m=+38.143873465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:57.807871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:57.807794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s9tjz"] Sep 11 12:14:58.118541 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.118503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" event={"ID":"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72","Type":"ContainerStarted","Data":"142d0765228f5a37091a1912010d89e2235c8fca81fd552e58151d29aad7287e"} Sep 11 12:14:58.120940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.120872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" event={"ID":"e8df42ea-beeb-4e7b-92a0-1d23cd64181c","Type":"ContainerStarted","Data":"7b220d2fb6c900fb711a5321da10668d7092bf8a2172048d5c6842ce2c29a72b"} Sep 11 12:14:58.122257 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.122211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" event={"ID":"b889fdfc-8f57-4240-abd7-1a1ed43366aa","Type":"ContainerStarted","Data":"b769f63cb3681ceb5ff9c126529cf414ebbca15dcf4e9ed7d42d2a93e00e3cab"} Sep 11 12:14:58.123777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.123753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" event={"ID":"e75bc421-07a1-49a9-b0e3-83fc5908fbae","Type":"ContainerStarted","Data":"29fe0ae8957c9b572f3657af6194ef1d079c38c1d403c211da07b9810c3a9c9c"} Sep 11 12:14:58.125068 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.125041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" event={"ID":"8a591965-0202-4194-b784-8aa97294024c","Type":"ContainerStarted","Data":"b96997fc6c56b9bf3be8cd339382297e59a3f613bd3abfc426bdd904f4d8a4bd"} Sep 11 12:14:58.564423 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.564391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:14:58.564602 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.564519 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:14:58.564694 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.564604 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.564581354 +0000 UTC m=+40.052786393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:14:58.665640 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.665591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:58.665790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.665673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:14:58.665790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:58.665710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:14:58.665908 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.665796 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.665774056 +0000 UTC m=+40.153979109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:14:58.665908 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.665828 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:14:58.665908 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.665866 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:58.665908 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.665871 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.665858701 +0000 UTC m=+40.154063758 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:14:58.666119 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:58.665933 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:15:00.665916489 +0000 UTC m=+40.154121526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:14:59.674088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:59.674052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:14:59.674212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:59.674216 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:59.674278 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:59.674292 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:59.674299 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:03.674279758 +0000 UTC m=+43.162484807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:14:59.674594 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:14:59.674324 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:03.674314057 +0000 UTC m=+43.162519093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:15:00.350794 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:00.350758 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0874441a_169d_48eb_9424_330a62809713.slice/crio-79f1bd1144ce138ba023b25a68c23ab4c619f8832b7f964aaf9096d40f8035e5 WatchSource:0}: Error finding container 79f1bd1144ce138ba023b25a68c23ab4c619f8832b7f964aaf9096d40f8035e5: Status 404 returned error can't find the container with id 79f1bd1144ce138ba023b25a68c23ab4c619f8832b7f964aaf9096d40f8035e5 Sep 11 12:15:00.581257 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:00.581221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:00.581838 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.581422 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:15:00.581838 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.581485 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:04.581470514 +0000 UTC m=+44.069675548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:00.682018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:00.682074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:00.682095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.682224 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.682278 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:15:04.682261079 +0000 UTC m=+44.170466119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.682690 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:04.682678761 +0000 UTC m=+44.170883795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.682745 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:15:00.682803 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:00.682775 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:04.682765116 +0000 UTC m=+44.170970156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:15:01.137510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:01.137403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s9tjz" event={"ID":"0874441a-169d-48eb-9424-330a62809713","Type":"ContainerStarted","Data":"79f1bd1144ce138ba023b25a68c23ab4c619f8832b7f964aaf9096d40f8035e5"} Sep 11 12:15:02.144659 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.144605 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="7c046b6264b31ab7709bef7e822086af5f58bd35ec37904afb87ac5d182cbfe0" exitCode=0 Sep 11 12:15:02.145091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.144706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"7c046b6264b31ab7709bef7e822086af5f58bd35ec37904afb87ac5d182cbfe0"} Sep 11 12:15:02.147415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.147390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" event={"ID":"e8df42ea-beeb-4e7b-92a0-1d23cd64181c","Type":"ContainerStarted","Data":"65c3b86e894f62790961687d03d33b3642d0f69d81dfa18879844a8c88f47a79"} Sep 11 12:15:02.149180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.149148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r9tvs" event={"ID":"ecdce906-182f-43e2-b55a-6269548d60b6","Type":"ContainerStarted","Data":"558cef2ad5048443a5df95af55f3d65ea808a78d0dccab820d34a131505e211f"} Sep 11 12:15:02.191838 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.191786 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-pnj8v" podStartSLOduration=2.022745573 podStartE2EDuration="6.191768081s" totalImagesPullingTime="4.169022508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:02.191272144 +0000 UTC m=+41.679477205" watchObservedRunningTime="2026-09-11 12:15:02.191768081 +0000 UTC m=+41.679973139" Sep 11 12:15:02.210768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:02.210721 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r9tvs" podStartSLOduration=2.6427762 podStartE2EDuration="7.210704207s" totalImagesPullingTime="4.567928007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:02.209707026 +0000 UTC m=+41.697912085" watchObservedRunningTime="2026-09-11 12:15:02.210704207 +0000 UTC m=+41.698909270" Sep 11 12:15:03.710008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:03.709982 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:03.710020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:03.710144 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:03.710162 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:03.710164 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:03.710231 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.710214239 +0000 UTC m=+51.198419274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:15:03.710350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:03.710243 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.710237491 +0000 UTC m=+51.198442526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:15:04.617155 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:04.617123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:04.617324 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.617295 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:15:04.617376 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.617358 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:12.617343346 +0000 UTC m=+52.105548382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:15:04.718269 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:04.718237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:04.718315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:04.718347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.718410 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:12.718391696 +0000 UTC m=+52.206596750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.718460 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.718498 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:15:12.71848761 +0000 UTC m=+52.206692647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.718557 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:15:04.718683 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:04.718595 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:12.718576224 +0000 UTC m=+52.206781259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:15:07.172064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.171952 2575 generic.go:356] "Generic (PLEG): container finished" podID="b6854465-c2ab-44f6-bd2b-cc6375df2348" containerID="0308dcb5b4cdd51b8a4f0b7805f1412c9d8d6409867f65a079fde0d28e94404e" exitCode=0 Sep 11 12:15:07.172064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.172046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerDied","Data":"0308dcb5b4cdd51b8a4f0b7805f1412c9d8d6409867f65a079fde0d28e94404e"} Sep 11 12:15:07.174018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.173999 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/0.log" Sep 11 12:15:07.174121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.174028 2575 generic.go:356] "Generic (PLEG): container finished" podID="8a591965-0202-4194-b784-8aa97294024c" containerID="97b839e621af7c97401827a7e39eaa3d0700ea5773959a7f6cecadedb8a30e8c" exitCode=255 Sep 11 12:15:07.174121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.174083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" event={"ID":"8a591965-0202-4194-b784-8aa97294024c","Type":"ContainerDied","Data":"97b839e621af7c97401827a7e39eaa3d0700ea5773959a7f6cecadedb8a30e8c"} Sep 11 12:15:07.174332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.174317 2575 scope.go:117] "RemoveContainer" containerID="97b839e621af7c97401827a7e39eaa3d0700ea5773959a7f6cecadedb8a30e8c" Sep 11 12:15:07.180787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.180761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" event={"ID":"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72","Type":"ContainerStarted","Data":"9c047e4fbe56b1308f4d1042fe6b2161e4c9b809695c6f007626ec5a16400a00"} Sep 11 12:15:07.182450 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.182426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s9tjz" event={"ID":"0874441a-169d-48eb-9424-330a62809713","Type":"ContainerStarted","Data":"664ad7b8aa3f5b2f93b36da8969bb761a8dfc95e61c4f51ab31f28a20d20f3f2"} Sep 11 12:15:07.184044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.184022 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" event={"ID":"b889fdfc-8f57-4240-abd7-1a1ed43366aa","Type":"ContainerStarted","Data":"0cd001ef71e4df0bd33251af3b9ca9e83ef800eeb4c6e57c603509b96c4db217"} Sep 11 12:15:07.185416 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.185400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" event={"ID":"e75bc421-07a1-49a9-b0e3-83fc5908fbae","Type":"ContainerStarted","Data":"5f261b46a23998a7ef6d426f91ad676426612c84b262992160062c8b78b3e250"} Sep 11 12:15:07.251008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.250092 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" podStartSLOduration=2.689438339 podStartE2EDuration="11.250070288s" totalImagesPullingTime="8.560631949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:07.225048427 +0000 UTC m=+46.713253481" watchObservedRunningTime="2026-09-11 12:15:07.250070288 +0000 UTC m=+46.738275346" Sep 11 12:15:07.275083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.275041 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-nqz29" podStartSLOduration=2.606170838 podStartE2EDuration="11.27502556s" totalImagesPullingTime="8.668854722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:07.272360194 +0000 UTC m=+46.760565252" watchObservedRunningTime="2026-09-11 12:15:07.27502556 +0000 UTC m=+46.763230616" Sep 11 12:15:07.291761 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.291699 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:07.291816 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.291778 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:07.294192 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.294146 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-s9tjz" podStartSLOduration=5.605003023 podStartE2EDuration="11.294128686s" totalImagesPullingTime="5.689125663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:07.293643967 +0000 UTC m=+46.781849025" watchObservedRunningTime="2026-09-11 12:15:07.294128686 +0000 UTC m=+46.782333744" Sep 11 12:15:07.311886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.311144 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" podStartSLOduration=2.699145968 podStartE2EDuration="11.311125043s" totalImagesPullingTime="8.611979075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:07.310236356 +0000 UTC m=+46.798441413" watchObservedRunningTime="2026-09-11 12:15:07.311125043 +0000 UTC m=+46.799330099" Sep 11 12:15:07.649423 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.649387 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4kxnc"] Sep 11 12:15:07.666972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.666938 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4kxnc"] Sep 11 12:15:07.667101 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.667080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.669560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.669526 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 12:15:07.669718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.669542 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 12:15:07.669774 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.669724 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l5m57\"" Sep 11 12:15:07.745712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.745681 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f653d-7336-4495-bff1-a098156d102c-crio-socket\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.745850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.745712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2slf\" (UniqueName: \"kubernetes.io/projected/e58f653d-7336-4495-bff1-a098156d102c-kube-api-access-g2slf\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.745850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.745744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.745971 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.745894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.745971 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.745934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f653d-7336-4495-bff1-a098156d102c-data-volume\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.846925 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.846902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f653d-7336-4495-bff1-a098156d102c-crio-socket\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.846931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2slf\" (UniqueName: \"kubernetes.io/projected/e58f653d-7336-4495-bff1-a098156d102c-kube-api-access-g2slf\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.846960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.847031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e58f653d-7336-4495-bff1-a098156d102c-crio-socket\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.847055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847208 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:07.847072 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 12:15:07.847208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.847092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f653d-7336-4495-bff1-a098156d102c-data-volume\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.847208 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:07.847125 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.347108451 +0000 UTC m=+47.835313504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : secret "insights-runtime-extractor-tls" not found Sep 11 12:15:07.847208 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:07.847181 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 12:15:07.847428 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:07.847235 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:08.34721893 +0000 UTC m=+47.835423969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : configmap "kube-rbac-proxy" not found Sep 11 12:15:07.847428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.847334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e58f653d-7336-4495-bff1-a098156d102c-data-volume\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:07.856355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:07.856334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2slf\" (UniqueName: \"kubernetes.io/projected/e58f653d-7336-4495-bff1-a098156d102c-kube-api-access-g2slf\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:08.193068 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.193034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" event={"ID":"b6854465-c2ab-44f6-bd2b-cc6375df2348","Type":"ContainerStarted","Data":"6e9ec819b9a7a735a9b93fe03563faa35d9cfdd411447ea24f4dcd428c341813"} Sep 11 12:15:08.194695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.194676 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/1.log" Sep 11 12:15:08.195153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.195138 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/0.log" Sep 11 12:15:08.195214 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.195171 2575 generic.go:356] "Generic (PLEG): container finished" podID="8a591965-0202-4194-b784-8aa97294024c" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" exitCode=255 Sep 11 12:15:08.195259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.195243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" event={"ID":"8a591965-0202-4194-b784-8aa97294024c","Type":"ContainerDied","Data":"894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6"} Sep 11 12:15:08.195312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.195285 2575 scope.go:117] "RemoveContainer" containerID="97b839e621af7c97401827a7e39eaa3d0700ea5773959a7f6cecadedb8a30e8c" Sep 11 12:15:08.195523 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.195494 2575 scope.go:117] "RemoveContainer" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" Sep 11 12:15:08.195731 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:08.195707 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dbnv2_openshift-console-operator(8a591965-0202-4194-b784-8aa97294024c)\"" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podUID="8a591965-0202-4194-b784-8aa97294024c" Sep 11 12:15:08.220842 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.217582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jgnzl" podStartSLOduration=12.502755959 podStartE2EDuration="47.217563698s" totalImagesPullingTime="34.714807739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:08.2162839 +0000 UTC m=+47.704488957" watchObservedRunningTime="2026-09-11 12:15:08.217563698 +0000 UTC m=+47.705768755" Sep 11 12:15:08.353128 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.353106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:08.353207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:08.353159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:08.353265 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:08.353227 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 12:15:08.353265 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:08.353258 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 12:15:08.353351 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:08.353291 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.353275482 +0000 UTC m=+48.841480516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : secret "insights-runtime-extractor-tls" not found Sep 11 12:15:08.353351 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:08.353306 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:09.353296527 +0000 UTC m=+48.841501562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : configmap "kube-rbac-proxy" not found Sep 11 12:15:09.201597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:09.201569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/1.log" Sep 11 12:15:09.201976 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:09.201879 2575 scope.go:117] "RemoveContainer" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" Sep 11 12:15:09.202038 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:09.202019 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dbnv2_openshift-console-operator(8a591965-0202-4194-b784-8aa97294024c)\"" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podUID="8a591965-0202-4194-b784-8aa97294024c" Sep 11 12:15:09.362419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:09.362386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:09.362562 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:09.362462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:09.362562 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:09.362532 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 12:15:09.362687 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:09.362590 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.362576363 +0000 UTC m=+50.850781398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : secret "insights-runtime-extractor-tls" not found Sep 11 12:15:09.362687 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:09.362632 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 12:15:09.362756 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:09.362714 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:11.362696872 +0000 UTC m=+50.850901907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : configmap "kube-rbac-proxy" not found Sep 11 12:15:09.424984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:09.424953 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5wrfm_b34698e3-de0b-4d89-8c31-901d9efea428/dns-node-resolver/0.log" Sep 11 12:15:10.824002 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:10.823976 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w4tw4_0ba65492-6ff4-489f-b229-3fe6896728aa/node-ca/0.log" Sep 11 12:15:11.378212 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:11.378166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:11.378406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:11.378249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:11.378406 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.378323 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 12:15:11.378406 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.378335 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 12:15:11.378406 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.378389 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.378374376 +0000 UTC m=+54.866579410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : configmap "kube-rbac-proxy" not found Sep 11 12:15:11.378575 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.378410 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:15.378394865 +0000 UTC m=+54.866599900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : secret "insights-runtime-extractor-tls" not found Sep 11 12:15:11.425569 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:11.425541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9tvs_ecdce906-182f-43e2-b55a-6269548d60b6/serve-healthcheck-canary/0.log" Sep 11 12:15:11.780682 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:11.780643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:11.780824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:11.780704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:11.780824 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.780801 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 12:15:11.780824 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.780815 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ff64577df-s6xw8: secret "image-registry-tls" not found Sep 11 12:15:11.780928 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.780863 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls podName:2e14ba71-3975-4253-a03a-aa566829d934 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:27.780849613 +0000 UTC m=+67.269054648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls") pod "image-registry-6ff64577df-s6xw8" (UID: "2e14ba71-3975-4253-a03a-aa566829d934") : secret "image-registry-tls" not found Sep 11 12:15:11.780928 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.780867 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 12:15:11.781000 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:11.780932 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls podName:17d41008-bde7-4c58-970c-840b80e7165b nodeName:}" failed. No retries permitted until 2026-09-11 12:15:27.780917703 +0000 UTC m=+67.269122737 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls") pod "dns-default-p5kc7" (UID: "17d41008-bde7-4c58-970c-840b80e7165b") : secret "dns-default-metrics-tls" not found Sep 11 12:15:12.689159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:12.689135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:12.689531 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.689268 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 12:15:12.689531 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.689325 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls podName:b3f07b0f-741e-4d75-bf20-8b2de6b0bb59 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:28.68931269 +0000 UTC m=+68.177517760 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-v67lc" (UID: "b3f07b0f-741e-4d75-bf20-8b2de6b0bb59") : secret "samples-operator-tls" not found Sep 11 12:15:12.789914 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:12.789889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:12.790042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:12.789940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:12.790042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:12.789961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:12.790156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.790043 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 12:15:12.790156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.790057 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:28.790038334 +0000 UTC m=+68.278243392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : configmap references non-existent config key: service-ca.crt Sep 11 12:15:12.790156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.790062 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:12.790156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.790104 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls podName:d26cd9a3-d26f-4edd-b61e-face2237376e nodeName:}" failed. No retries permitted until 2026-09-11 12:15:28.790091887 +0000 UTC m=+68.278296922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrk7z" (UID: "d26cd9a3-d26f-4edd-b61e-face2237376e") : secret "cluster-monitoring-operator-tls" not found Sep 11 12:15:12.790156 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:12.790137 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs podName:29c80efb-0d0b-44ba-b8ec-95d061729afa nodeName:}" failed. No retries permitted until 2026-09-11 12:15:28.790120945 +0000 UTC m=+68.278326016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs") pod "router-default-68f4bb8d9b-qnbg7" (UID: "29c80efb-0d0b-44ba-b8ec-95d061729afa") : secret "router-metrics-certs-default" not found Sep 11 12:15:15.411512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:15.411472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:15.412125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:15.411545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:15.412125 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:15.411662 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 12:15:15.412125 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:15.411738 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls podName:e58f653d-7336-4495-bff1-a098156d102c nodeName:}" failed. No retries permitted until 2026-09-11 12:15:23.411721491 +0000 UTC m=+62.899926526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4kxnc" (UID: "e58f653d-7336-4495-bff1-a098156d102c") : secret "insights-runtime-extractor-tls" not found Sep 11 12:15:15.412540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:15.412253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e58f653d-7336-4495-bff1-a098156d102c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:17.291939 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:17.291892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:17.292324 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:17.291963 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:17.292361 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:17.292348 2575 scope.go:117] "RemoveContainer" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" Sep 11 12:15:17.292544 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:17.292520 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dbnv2_openshift-console-operator(8a591965-0202-4194-b784-8aa97294024c)\"" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podUID="8a591965-0202-4194-b784-8aa97294024c" Sep 11 12:15:21.109566 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:21.109527 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wqf8h" Sep 11 12:15:23.478125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:23.478088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:23.480811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:23.480778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e58f653d-7336-4495-bff1-a098156d102c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4kxnc\" (UID: \"e58f653d-7336-4495-bff1-a098156d102c\") " pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:23.580138 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:23.580106 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l5m57\"" Sep 11 12:15:23.588592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:23.588565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4kxnc" Sep 11 12:15:23.862560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:23.862491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4kxnc"] Sep 11 12:15:23.866762 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:23.866722 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58f653d_7336_4495_bff1_a098156d102c.slice/crio-43b76285ee24f1b06e71b8faeae165feaf9d8d2c5852c97b5752d210252eae03 WatchSource:0}: Error finding container 43b76285ee24f1b06e71b8faeae165feaf9d8d2c5852c97b5752d210252eae03: Status 404 returned error can't find the container with id 43b76285ee24f1b06e71b8faeae165feaf9d8d2c5852c97b5752d210252eae03 Sep 11 12:15:24.258183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:24.258143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kxnc" event={"ID":"e58f653d-7336-4495-bff1-a098156d102c","Type":"ContainerStarted","Data":"0fa9547b10c1ce117ed96fb59805eaaad0f7f550ac15a29d0fadeb69fe559a67"} Sep 11 12:15:24.258342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:24.258232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kxnc" event={"ID":"e58f653d-7336-4495-bff1-a098156d102c","Type":"ContainerStarted","Data":"43b76285ee24f1b06e71b8faeae165feaf9d8d2c5852c97b5752d210252eae03"} Sep 11 12:15:25.263159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.263132 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kxnc" event={"ID":"e58f653d-7336-4495-bff1-a098156d102c","Type":"ContainerStarted","Data":"0dd4ae7ba19ae028244d75ea781d5989f6d50f1e8f9d854d9b17546293967565"} Sep 11 12:15:25.697858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.697828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:15:25.700841 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.700819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 12:15:25.711575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.711553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c7c065-ce6f-4bc4-a44b-5e00d23b880b-metrics-certs\") pod \"network-metrics-daemon-9mtmd\" (UID: \"34c7c065-ce6f-4bc4-a44b-5e00d23b880b\") " pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:15:25.799138 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.799112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:15:25.799278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.799253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:15:25.802739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.802574 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 12:15:25.802739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.802683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhrkx\" (UniqueName: \"kubernetes.io/projected/5a7a660c-88c9-4eea-bc99-fc47ba8675f1-kube-api-access-jhrkx\") pod \"network-check-target-g6vhr\" (UID: \"5a7a660c-88c9-4eea-bc99-fc47ba8675f1\") " pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:15:25.812595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.812575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b2b3ef44-c676-4dbe-bd64-17e7a276afee-original-pull-secret\") pod \"global-pull-secret-syncer-glbfl\" (UID: \"b2b3ef44-c676-4dbe-bd64-17e7a276afee\") " pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:15:25.978198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.977967 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-glbfl" Sep 11 12:15:25.988186 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.988163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dhjsw\"" Sep 11 12:15:25.992487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.992470 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jvbml\"" Sep 11 12:15:25.995435 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:25.995416 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:15:26.001004 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:26.000986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9mtmd" Sep 11 12:15:26.622338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:26.622234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9mtmd"] Sep 11 12:15:26.625810 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:26.625781 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c7c065_ce6f_4bc4_a44b_5e00d23b880b.slice/crio-e78a63d47c4c7f63c09fbb39034423cc905c10e0dc091705e70ea62b9b783608 WatchSource:0}: Error finding container e78a63d47c4c7f63c09fbb39034423cc905c10e0dc091705e70ea62b9b783608: Status 404 returned error can't find the container with id e78a63d47c4c7f63c09fbb39034423cc905c10e0dc091705e70ea62b9b783608 Sep 11 12:15:26.647423 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:26.647404 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g6vhr"] Sep 11 12:15:26.649597 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:26.649578 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7a660c_88c9_4eea_bc99_fc47ba8675f1.slice/crio-7db59eb6d6ac14de902c4b3d2276f1574008eebd97c4c30760039a3d3f1196fe WatchSource:0}: Error finding container 7db59eb6d6ac14de902c4b3d2276f1574008eebd97c4c30760039a3d3f1196fe: Status 404 returned error can't find the container with id 7db59eb6d6ac14de902c4b3d2276f1574008eebd97c4c30760039a3d3f1196fe Sep 11 12:15:26.664379 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:26.664354 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-glbfl"] Sep 11 12:15:26.667179 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:26.667153 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b3ef44_c676_4dbe_bd64_17e7a276afee.slice/crio-25906a815a4073a2aba2c86c134b715d0356c1c8b23373c341299f5831eaae9c WatchSource:0}: Error finding container 25906a815a4073a2aba2c86c134b715d0356c1c8b23373c341299f5831eaae9c: Status 404 returned error can't find the container with id 25906a815a4073a2aba2c86c134b715d0356c1c8b23373c341299f5831eaae9c Sep 11 12:15:27.272838 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.272795 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4kxnc" event={"ID":"e58f653d-7336-4495-bff1-a098156d102c","Type":"ContainerStarted","Data":"5af119ed8417a7517287b71f463e688285647d1f54cf351e5e63afab7b2c1b4a"} Sep 11 12:15:27.274229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.274200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glbfl" event={"ID":"b2b3ef44-c676-4dbe-bd64-17e7a276afee","Type":"ContainerStarted","Data":"25906a815a4073a2aba2c86c134b715d0356c1c8b23373c341299f5831eaae9c"} Sep 11 12:15:27.276004 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.275975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6vhr" event={"ID":"5a7a660c-88c9-4eea-bc99-fc47ba8675f1","Type":"ContainerStarted","Data":"a34e463c77d85b1609f04156e44decda320d1cc9da8727b6d0ef156e2f1cda29"} Sep 11 12:15:27.276116 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.276020 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g6vhr" event={"ID":"5a7a660c-88c9-4eea-bc99-fc47ba8675f1","Type":"ContainerStarted","Data":"7db59eb6d6ac14de902c4b3d2276f1574008eebd97c4c30760039a3d3f1196fe"} Sep 11 12:15:27.276116 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.276071 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:15:27.277456 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.277431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9mtmd" event={"ID":"34c7c065-ce6f-4bc4-a44b-5e00d23b880b","Type":"ContainerStarted","Data":"e78a63d47c4c7f63c09fbb39034423cc905c10e0dc091705e70ea62b9b783608"} Sep 11 12:15:27.317967 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.317923 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4kxnc" podStartSLOduration=17.93392479 podStartE2EDuration="20.317907611s" totalImagesPullingTime="2.383982821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:27.298120322 +0000 UTC m=+66.786325378" watchObservedRunningTime="2026-09-11 12:15:27.317907611 +0000 UTC m=+66.806112668" Sep 11 12:15:27.318086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.318066 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g6vhr" podStartSLOduration=66.318062312 podStartE2EDuration="1m6.318062312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:27.317884154 +0000 UTC m=+66.806089210" watchObservedRunningTime="2026-09-11 12:15:27.318062312 +0000 UTC m=+66.806267368" Sep 11 12:15:27.820232 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.820188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:27.820232 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.820239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:27.823932 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.823911 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17d41008-bde7-4c58-970c-840b80e7165b-metrics-tls\") pod \"dns-default-p5kc7\" (UID: \"17d41008-bde7-4c58-970c-840b80e7165b\") " pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:27.824124 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.824105 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"image-registry-6ff64577df-s6xw8\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:27.947291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:27.947267 2575 scope.go:117] "RemoveContainer" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" Sep 11 12:15:28.039361 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.039337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bmqgw\"" Sep 11 12:15:28.046998 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.046976 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:28.053118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.053096 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lgchp\"" Sep 11 12:15:28.060142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.059968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:28.288270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.288189 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.289858 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/1.log" Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.289894 2575 generic.go:356] "Generic (PLEG): container finished" podID="8a591965-0202-4194-b784-8aa97294024c" containerID="f3eda6c135817746faad738f5a8f5bdd00beca76c5af1257d229277404ae2f07" exitCode=255 Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.290483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" event={"ID":"8a591965-0202-4194-b784-8aa97294024c","Type":"ContainerDied","Data":"f3eda6c135817746faad738f5a8f5bdd00beca76c5af1257d229277404ae2f07"} Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.290536 2575 scope.go:117] "RemoveContainer" containerID="894c620673cbccdae4fa7249d23354bb67312d3cd70f8fc476f9d1e6039b20f6" Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.291456 2575 scope.go:117] "RemoveContainer" containerID="f3eda6c135817746faad738f5a8f5bdd00beca76c5af1257d229277404ae2f07" Sep 11 12:15:28.292089 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:28.291630 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-dbnv2_openshift-console-operator(8a591965-0202-4194-b784-8aa97294024c)\"" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podUID="8a591965-0202-4194-b784-8aa97294024c" Sep 11 12:15:28.534643 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.534451 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:15:28.538567 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:28.538537 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e14ba71_3975_4253_a03a_aa566829d934.slice/crio-a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d WatchSource:0}: Error finding container a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d: Status 404 returned error can't find the container with id a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d Sep 11 12:15:28.549657 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.549634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p5kc7"] Sep 11 12:15:28.550668 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:28.550647 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d41008_bde7_4c58_970c_840b80e7165b.slice/crio-1fbc607329fe4efecbcb6ab20eea9f09924a797221ceb92ccb1f19918482d4ff WatchSource:0}: Error finding container 1fbc607329fe4efecbcb6ab20eea9f09924a797221ceb92ccb1f19918482d4ff: Status 404 returned error can't find the container with id 1fbc607329fe4efecbcb6ab20eea9f09924a797221ceb92ccb1f19918482d4ff Sep 11 12:15:28.729939 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.729908 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:28.732787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.732756 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b3f07b0f-741e-4d75-bf20-8b2de6b0bb59-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-v67lc\" (UID: \"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:28.831275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.831208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:28.831275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.831261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:28.831776 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.831281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:28.831931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.831903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29c80efb-0d0b-44ba-b8ec-95d061729afa-service-ca-bundle\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:28.834491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.834470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d26cd9a3-d26f-4edd-b61e-face2237376e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrk7z\" (UID: \"d26cd9a3-d26f-4edd-b61e-face2237376e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:28.834813 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.834793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29c80efb-0d0b-44ba-b8ec-95d061729afa-metrics-certs\") pod \"router-default-68f4bb8d9b-qnbg7\" (UID: \"29c80efb-0d0b-44ba-b8ec-95d061729afa\") " pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:28.870977 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.870952 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w5whl\"" Sep 11 12:15:28.877777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:28.877761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" Sep 11 12:15:29.023513 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.023480 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ljcfs\"" Sep 11 12:15:29.031306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.031278 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:29.033262 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.033245 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rps94\"" Sep 11 12:15:29.040567 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.040553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" Sep 11 12:15:29.164316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.164286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc"] Sep 11 12:15:29.295524 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.295503 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:15:29.297527 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.297493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" event={"ID":"2e14ba71-3975-4253-a03a-aa566829d934","Type":"ContainerStarted","Data":"fb27028990654777a23a1c353ded1283d9f833939e87c494d3cb1a44b5a6053a"} Sep 11 12:15:29.297668 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.297554 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:29.297668 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.297578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" event={"ID":"2e14ba71-3975-4253-a03a-aa566829d934","Type":"ContainerStarted","Data":"a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d"} Sep 11 12:15:29.299129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.299095 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" event={"ID":"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59","Type":"ContainerStarted","Data":"469bc804864e68db64571ae4530331e6d6b9c9646847bf681201586301e5b0d9"} Sep 11 12:15:29.300433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.300407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p5kc7" event={"ID":"17d41008-bde7-4c58-970c-840b80e7165b","Type":"ContainerStarted","Data":"1fbc607329fe4efecbcb6ab20eea9f09924a797221ceb92ccb1f19918482d4ff"} Sep 11 12:15:29.302575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.302550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9mtmd" event={"ID":"34c7c065-ce6f-4bc4-a44b-5e00d23b880b","Type":"ContainerStarted","Data":"9efb1c42cf171a115dc79f647873bc008a408815f6e2eecbc07d6285fc1ada71"} Sep 11 12:15:29.302696 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.302600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9mtmd" event={"ID":"34c7c065-ce6f-4bc4-a44b-5e00d23b880b","Type":"ContainerStarted","Data":"1042a52ad2ca84a9a6f37a693cc1034237be02a2dde3996c306ccc7e3e283077"} Sep 11 12:15:29.325542 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.325523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68f4bb8d9b-qnbg7"] Sep 11 12:15:29.329907 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:29.329887 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c80efb_0d0b_44ba_b8ec_95d061729afa.slice/crio-e0bae278ced5e38bfc4ff4dc6c90a053e07369dc52780bb42507e2a9b659f22d WatchSource:0}: Error finding container e0bae278ced5e38bfc4ff4dc6c90a053e07369dc52780bb42507e2a9b659f22d: Status 404 returned error can't find the container with id e0bae278ced5e38bfc4ff4dc6c90a053e07369dc52780bb42507e2a9b659f22d Sep 11 12:15:29.344335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.344289 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" podStartSLOduration=68.344269333 podStartE2EDuration="1m8.344269333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:29.327989069 +0000 UTC m=+68.816194127" watchObservedRunningTime="2026-09-11 12:15:29.344269333 +0000 UTC m=+68.832474390" Sep 11 12:15:29.345023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.344997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z"] Sep 11 12:15:29.348050 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:29.348020 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26cd9a3_d26f_4edd_b61e_face2237376e.slice/crio-b9075664e29a8a8fa2d76a7464195524e648faa093b8103b0b85efaa9b5dd1af WatchSource:0}: Error finding container b9075664e29a8a8fa2d76a7464195524e648faa093b8103b0b85efaa9b5dd1af: Status 404 returned error can't find the container with id b9075664e29a8a8fa2d76a7464195524e648faa093b8103b0b85efaa9b5dd1af Sep 11 12:15:29.350895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:29.350759 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9mtmd" podStartSLOduration=66.709985374 podStartE2EDuration="1m8.350740726s" totalImagesPullingTime="1.640755352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:29.350122762 +0000 UTC m=+68.838327818" watchObservedRunningTime="2026-09-11 12:15:29.350740726 +0000 UTC m=+68.838945785" Sep 11 12:15:30.309706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:30.309667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" event={"ID":"29c80efb-0d0b-44ba-b8ec-95d061729afa","Type":"ContainerStarted","Data":"f9fa3fb48ae62487795076bd8cf92071ba7e88874ca031f99fcca3fe7908e6d3"} Sep 11 12:15:30.310099 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:30.309730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" event={"ID":"29c80efb-0d0b-44ba-b8ec-95d061729afa","Type":"ContainerStarted","Data":"e0bae278ced5e38bfc4ff4dc6c90a053e07369dc52780bb42507e2a9b659f22d"} Sep 11 12:15:30.311714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:30.311651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" event={"ID":"d26cd9a3-d26f-4edd-b61e-face2237376e","Type":"ContainerStarted","Data":"b9075664e29a8a8fa2d76a7464195524e648faa093b8103b0b85efaa9b5dd1af"} Sep 11 12:15:30.368539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:30.368494 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" podStartSLOduration=34.368477619 podStartE2EDuration="34.368477619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:30.366792471 +0000 UTC m=+69.854997530" watchObservedRunningTime="2026-09-11 12:15:30.368477619 +0000 UTC m=+69.856682670" Sep 11 12:15:31.031953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:31.031921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:31.035316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:31.035292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:31.317368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:31.317281 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:31.318752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:31.318727 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68f4bb8d9b-qnbg7" Sep 11 12:15:33.326790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:33.326746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-glbfl" event={"ID":"b2b3ef44-c676-4dbe-bd64-17e7a276afee","Type":"ContainerStarted","Data":"1a43c10bdc0be3896b7bae5cc85c3f7d44c2d9d0e4045b8a8e51d5ae4ad5ffc3"} Sep 11 12:15:33.328931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:33.328896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p5kc7" event={"ID":"17d41008-bde7-4c58-970c-840b80e7165b","Type":"ContainerStarted","Data":"11d15412733fe04e07ce685466316c7a0c1be4d2d004d5a622480f442c123a58"} Sep 11 12:15:33.329068 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:33.328958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p5kc7" event={"ID":"17d41008-bde7-4c58-970c-840b80e7165b","Type":"ContainerStarted","Data":"0942ef427eb19ca936176aee77adc64529095224b48ed49fbf3e987680a877e9"} Sep 11 12:15:33.347422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:33.347376 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-glbfl" podStartSLOduration=66.197775954 podStartE2EDuration="1m12.347361228s" totalImagesPullingTime="6.149585274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:33.345216866 +0000 UTC m=+72.833421927" watchObservedRunningTime="2026-09-11 12:15:33.347361228 +0000 UTC m=+72.835566286" Sep 11 12:15:33.370849 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:33.370783 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p5kc7" podStartSLOduration=34.105443262 podStartE2EDuration="38.37076843s" totalImagesPullingTime="4.265325168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:33.368649515 +0000 UTC m=+72.856854572" watchObservedRunningTime="2026-09-11 12:15:33.37076843 +0000 UTC m=+72.858973488" Sep 11 12:15:34.335221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.335182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" event={"ID":"d26cd9a3-d26f-4edd-b61e-face2237376e","Type":"ContainerStarted","Data":"e2dec79b4c41930b8e24a97993e98aa484ea9721dc88bf771144b3e3ee96924c"} Sep 11 12:15:34.337178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.337152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" event={"ID":"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59","Type":"ContainerStarted","Data":"d0825bdb6fa97ed1f89e660a7f1af8630bdc28984b0514f569bf1c294a5748ea"} Sep 11 12:15:34.337311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.337212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" event={"ID":"b3f07b0f-741e-4d75-bf20-8b2de6b0bb59","Type":"ContainerStarted","Data":"6b5f2e70477108acc2c6e7f5375a7068b7267827df8c775074815afa81b9b14d"} Sep 11 12:15:34.337464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.337444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:34.354432 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.354339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrk7z" podStartSLOduration=33.603102088 podStartE2EDuration="38.354327553s" totalImagesPullingTime="4.751225465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:34.353834753 +0000 UTC m=+73.842039811" watchObservedRunningTime="2026-09-11 12:15:34.354327553 +0000 UTC m=+73.842532609" Sep 11 12:15:34.375449 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:34.375408 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-v67lc" podStartSLOduration=34.488886531 podStartE2EDuration="38.375395483s" totalImagesPullingTime="3.886508952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:34.374792111 +0000 UTC m=+73.862997185" watchObservedRunningTime="2026-09-11 12:15:34.375395483 +0000 UTC m=+73.863600563" Sep 11 12:15:37.291936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:37.291900 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:37.292528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:37.291966 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:37.292528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:37.292363 2575 scope.go:117] "RemoveContainer" containerID="f3eda6c135817746faad738f5a8f5bdd00beca76c5af1257d229277404ae2f07" Sep 11 12:15:37.292699 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:37.292571 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-dbnv2_openshift-console-operator(8a591965-0202-4194-b784-8aa97294024c)\"" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podUID="8a591965-0202-4194-b784-8aa97294024c" Sep 11 12:15:40.805220 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.805184 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t549c"] Sep 11 12:15:40.821320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.821295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:40.825301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.825275 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 12:15:40.825469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.825322 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 12:15:40.825469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.825282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xf5h2\"" Sep 11 12:15:40.825641 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.825470 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 12:15:40.825641 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.825546 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t549c"] Sep 11 12:15:40.914558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.914532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:40.914702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.914597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e96ec-dd41-41a2-8180-44689fc76046-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:40.914702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.914659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:40.914702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:40.914688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckf4j\" (UniqueName: \"kubernetes.io/projected/7e3e96ec-dd41-41a2-8180-44689fc76046-kube-api-access-ckf4j\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.014909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.014890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.015019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.014935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e96ec-dd41-41a2-8180-44689fc76046-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.015019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.014955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.015019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.014981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckf4j\" (UniqueName: \"kubernetes.io/projected/7e3e96ec-dd41-41a2-8180-44689fc76046-kube-api-access-ckf4j\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.015662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.015645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e3e96ec-dd41-41a2-8180-44689fc76046-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.017308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.017284 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.017403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.017384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7e3e96ec-dd41-41a2-8180-44689fc76046-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.024902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.024884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckf4j\" (UniqueName: \"kubernetes.io/projected/7e3e96ec-dd41-41a2-8180-44689fc76046-kube-api-access-ckf4j\") pod \"prometheus-operator-68bcd7f987-t549c\" (UID: \"7e3e96ec-dd41-41a2-8180-44689fc76046\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.132792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.132735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" Sep 11 12:15:41.404180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:41.404154 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t549c"] Sep 11 12:15:42.375156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:42.375120 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" event={"ID":"7e3e96ec-dd41-41a2-8180-44689fc76046","Type":"ContainerStarted","Data":"733c934b2ebdddd2f40c2661219bd1c46c0d8067f63c3d33c1a2effd4fff3534"} Sep 11 12:15:43.383862 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:43.383779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" event={"ID":"7e3e96ec-dd41-41a2-8180-44689fc76046","Type":"ContainerStarted","Data":"0edabe3200a902d369242298723185efb70c57901808ae28111c01baa541677f"} Sep 11 12:15:43.383862 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:43.383827 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" event={"ID":"7e3e96ec-dd41-41a2-8180-44689fc76046","Type":"ContainerStarted","Data":"05244a8464b534ae9c08e5a5fc6e394b5c119906070b00faab8b38acdf450867"} Sep 11 12:15:43.421902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:43.421851 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t549c" podStartSLOduration=1.698858311 podStartE2EDuration="3.421831415s" totalImagesPullingTime="1.722973104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:43.42048497 +0000 UTC m=+82.908690038" watchObservedRunningTime="2026-09-11 12:15:43.421831415 +0000 UTC m=+82.910036475" Sep 11 12:15:44.343822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:44.343786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p5kc7" Sep 11 12:15:45.227230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.227200 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ccxfk"] Sep 11 12:15:45.229450 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.229434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.232807 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.232788 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 12:15:45.233160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.233143 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 12:15:45.233255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.233143 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bv4vc\"" Sep 11 12:15:45.233366 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.233293 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 12:15:45.239268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.239246 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj"] Sep 11 12:15:45.241165 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.241149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.243724 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.243706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 12:15:45.244258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.244244 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7j4xx\"" Sep 11 12:15:45.244292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.244276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 12:15:45.244474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.244459 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 12:15:45.255407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.255386 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj"] Sep 11 12:15:45.347690 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-textfile\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-metrics-client-ca\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9nv\" (UniqueName: \"kubernetes.io/projected/7e05a630-d3a8-413c-8fba-ac81f89b3775-kube-api-access-gb9nv\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-root\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.347916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-sys\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-wtmp\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.347916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347951 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.347990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.348005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bgb\" (UniqueName: \"kubernetes.io/projected/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-api-access-c9bgb\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.348070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.348033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-sys\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-wtmp\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bgb\" (UniqueName: \"kubernetes.io/projected/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-api-access-c9bgb\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.449314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449277 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-sys\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-wtmp\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:45.449354 2575 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449375 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-textfile\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:45.449427 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls podName:a54e1906-a3d1-4df9-a8a1-e20d87ae53cb nodeName:}" failed. No retries permitted until 2026-09-11 12:15:45.949407098 +0000 UTC m=+85.437612133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-db9vj" (UID: "a54e1906-a3d1-4df9-a8a1-e20d87ae53cb") : secret "kube-state-metrics-tls" not found Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:45.449480 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:45.449540 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls podName:7e05a630-d3a8-413c-8fba-ac81f89b3775 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:45.949523926 +0000 UTC m=+85.437728967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls") pod "node-exporter-ccxfk" (UID: "7e05a630-d3a8-413c-8fba-ac81f89b3775") : secret "node-exporter-tls" not found Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-metrics-client-ca\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9nv\" (UniqueName: \"kubernetes.io/projected/7e05a630-d3a8-413c-8fba-ac81f89b3775-kube-api-access-gb9nv\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-root\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.449739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.450267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.450267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.449958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-textfile\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.450267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.450014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7e05a630-d3a8-413c-8fba-ac81f89b3775-root\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.450267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.450188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-accelerators-collector-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.450267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.450243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e05a630-d3a8-413c-8fba-ac81f89b3775-metrics-client-ca\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.450492 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.450470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.450573 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.450556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.451800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.451782 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.451990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.451973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.460174 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.460147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9nv\" (UniqueName: \"kubernetes.io/projected/7e05a630-d3a8-413c-8fba-ac81f89b3775-kube-api-access-gb9nv\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.460174 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.460164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bgb\" (UniqueName: \"kubernetes.io/projected/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-api-access-c9bgb\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.953406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.953376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:45.953546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.953418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.955974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.955949 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7e05a630-d3a8-413c-8fba-ac81f89b3775-node-exporter-tls\") pod \"node-exporter-ccxfk\" (UID: \"7e05a630-d3a8-413c-8fba-ac81f89b3775\") " pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:45.956154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:45.956067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a54e1906-a3d1-4df9-a8a1-e20d87ae53cb-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-db9vj\" (UID: \"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:46.137895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:46.137863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ccxfk" Sep 11 12:15:46.148541 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:46.148520 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e05a630_d3a8_413c_8fba_ac81f89b3775.slice/crio-158ba3d73275fa4a580547bf2205863ff0b9b90369471511ac55ec2cf4ff8e9c WatchSource:0}: Error finding container 158ba3d73275fa4a580547bf2205863ff0b9b90369471511ac55ec2cf4ff8e9c: Status 404 returned error can't find the container with id 158ba3d73275fa4a580547bf2205863ff0b9b90369471511ac55ec2cf4ff8e9c Sep 11 12:15:46.149259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:46.149238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" Sep 11 12:15:46.396058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:46.396025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccxfk" event={"ID":"7e05a630-d3a8-413c-8fba-ac81f89b3775","Type":"ContainerStarted","Data":"158ba3d73275fa4a580547bf2205863ff0b9b90369471511ac55ec2cf4ff8e9c"} Sep 11 12:15:46.435507 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:46.435477 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj"] Sep 11 12:15:46.438010 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:46.437970 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54e1906_a3d1_4df9_a8a1_e20d87ae53cb.slice/crio-fa862040d7e3a86de0dadbd93d2b0e96c0a5db43a9347a6f31de4ad780955d8f WatchSource:0}: Error finding container fa862040d7e3a86de0dadbd93d2b0e96c0a5db43a9347a6f31de4ad780955d8f: Status 404 returned error can't find the container with id fa862040d7e3a86de0dadbd93d2b0e96c0a5db43a9347a6f31de4ad780955d8f Sep 11 12:15:47.402452 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:47.402423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccxfk" event={"ID":"7e05a630-d3a8-413c-8fba-ac81f89b3775","Type":"ContainerStarted","Data":"c25172cc56e9d629a02419f0541ffe12d89dd29845a53bbc3e0c903b0a36e94c"} Sep 11 12:15:47.403839 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:47.403813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" event={"ID":"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb","Type":"ContainerStarted","Data":"fa862040d7e3a86de0dadbd93d2b0e96c0a5db43a9347a6f31de4ad780955d8f"} Sep 11 12:15:48.053100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.053056 2575 patch_prober.go:28] interesting pod/image-registry-6ff64577df-s6xw8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 12:15:48.053263 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.053128 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" podUID="2e14ba71-3975-4253-a03a-aa566829d934" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:15:48.412545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.412514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" event={"ID":"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb","Type":"ContainerStarted","Data":"584f13961b406ae8fe16851f15cfe2fa1eae279647bdb0e3e361027919f94173"} Sep 11 12:15:48.413003 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.412623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" event={"ID":"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb","Type":"ContainerStarted","Data":"3e62ca0d10810270fb40f3a83a27b218d88a1ca36167777126a814bd592d8295"} Sep 11 12:15:48.413003 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.412658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" event={"ID":"a54e1906-a3d1-4df9-a8a1-e20d87ae53cb","Type":"ContainerStarted","Data":"1e7eab36680af9409289498d0e2af36c06493d9a8eaced5eb6f46dcbe42ebe0f"} Sep 11 12:15:48.414173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.414143 2575 generic.go:356] "Generic (PLEG): container finished" podID="7e05a630-d3a8-413c-8fba-ac81f89b3775" containerID="c25172cc56e9d629a02419f0541ffe12d89dd29845a53bbc3e0c903b0a36e94c" exitCode=0 Sep 11 12:15:48.414281 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.414203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccxfk" event={"ID":"7e05a630-d3a8-413c-8fba-ac81f89b3775","Type":"ContainerDied","Data":"c25172cc56e9d629a02419f0541ffe12d89dd29845a53bbc3e0c903b0a36e94c"} Sep 11 12:15:48.441359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.441294 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-db9vj" podStartSLOduration=1.752935213 podStartE2EDuration="3.441273153s" totalImagesPullingTime="1.68833794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:48.440528327 +0000 UTC m=+87.928733386" watchObservedRunningTime="2026-09-11 12:15:48.441273153 +0000 UTC m=+87.929478210" Sep 11 12:15:48.947159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:48.947129 2575 scope.go:117] "RemoveContainer" containerID="f3eda6c135817746faad738f5a8f5bdd00beca76c5af1257d229277404ae2f07" Sep 11 12:15:49.420502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.420459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccxfk" event={"ID":"7e05a630-d3a8-413c-8fba-ac81f89b3775","Type":"ContainerStarted","Data":"9726babdae549fd048e4ab0903cbc9ee1ff0e1fe37ed4ecfa24b7f79c77aa6a0"} Sep 11 12:15:49.420929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.420523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ccxfk" event={"ID":"7e05a630-d3a8-413c-8fba-ac81f89b3775","Type":"ContainerStarted","Data":"7cf7e018ed5702a512b444c9e9251407ebf8542a518af8b2c070833a57dc2966"} Sep 11 12:15:49.422268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.422252 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:15:49.422356 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.422300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" event={"ID":"8a591965-0202-4194-b784-8aa97294024c","Type":"ContainerStarted","Data":"f42c61a4b173f4818851605be13af8152447dd7e41e66306f6ac6252f1805d87"} Sep 11 12:15:49.422759 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.422743 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:49.444742 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.444698 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ccxfk" podStartSLOduration=3.258736126 podStartE2EDuration="4.444683746s" totalImagesPullingTime="1.115826474s" totalInitContainerRuntime="70.121146ms" isStatefulPod=true podCreationTimestamp="2026-09-11 12:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:49.44367638 +0000 UTC m=+88.931881438" watchObservedRunningTime="2026-09-11 12:15:49.444683746 +0000 UTC m=+88.932888803" Sep 11 12:15:49.564911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.564887 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" Sep 11 12:15:49.587183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.587134 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-dbnv2" podStartSLOduration=45.070034347 podStartE2EDuration="53.587115695s" totalImagesPullingTime="8.517081348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:49.465887839 +0000 UTC m=+88.954092898" watchObservedRunningTime="2026-09-11 12:15:49.587115695 +0000 UTC m=+89.075320753" Sep 11 12:15:49.758721 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.758628 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-rct9g"] Sep 11 12:15:49.760929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.760908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:15:49.763550 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.763520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 12:15:49.763666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.763520 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lkcgc\"" Sep 11 12:15:49.763835 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.763822 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 12:15:49.775843 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.775823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rct9g"] Sep 11 12:15:49.787038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.787007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr24d\" (UniqueName: \"kubernetes.io/projected/1a3738d8-352f-430d-8336-91b2319e5f1f-kube-api-access-rr24d\") pod \"downloads-76f99584f4-rct9g\" (UID: \"1a3738d8-352f-430d-8336-91b2319e5f1f\") " pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:15:49.887780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.887759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr24d\" (UniqueName: \"kubernetes.io/projected/1a3738d8-352f-430d-8336-91b2319e5f1f-kube-api-access-rr24d\") pod \"downloads-76f99584f4-rct9g\" (UID: \"1a3738d8-352f-430d-8336-91b2319e5f1f\") " pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:15:49.896481 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.896437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr24d\" (UniqueName: \"kubernetes.io/projected/1a3738d8-352f-430d-8336-91b2319e5f1f-kube-api-access-rr24d\") pod \"downloads-76f99584f4-rct9g\" (UID: \"1a3738d8-352f-430d-8336-91b2319e5f1f\") " pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:15:49.958580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.958552 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp"] Sep 11 12:15:49.960909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.960887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:49.963399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.963372 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 12:15:49.963503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.963432 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-957lg\"" Sep 11 12:15:49.974336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.974316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp"] Sep 11 12:15:49.988428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.988406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpj8m\" (UniqueName: \"kubernetes.io/projected/e848fd29-3e88-4e06-939d-2b3f37588c79-kube-api-access-gpj8m\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:49.988549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:49.988503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.070863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.070805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:15:50.089106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.089085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.089208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.089144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpj8m\" (UniqueName: \"kubernetes.io/projected/e848fd29-3e88-4e06-939d-2b3f37588c79-kube-api-access-gpj8m\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.089269 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:50.089210 2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 11 12:15:50.089315 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:15:50.089284 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert podName:e848fd29-3e88-4e06-939d-2b3f37588c79 nodeName:}" failed. No retries permitted until 2026-09-11 12:15:50.589263595 +0000 UTC m=+90.077468649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-wvltp" (UID: "e848fd29-3e88-4e06-939d-2b3f37588c79") : secret "monitoring-plugin-cert" not found Sep 11 12:15:50.098703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.098684 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpj8m\" (UniqueName: \"kubernetes.io/projected/e848fd29-3e88-4e06-939d-2b3f37588c79-kube-api-access-gpj8m\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.315930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.315905 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:15:50.349773 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:50.349745 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3738d8_352f_430d_8336_91b2319e5f1f.slice/crio-b29190a22bb89d0cd73c0b4f5a09c20ea956d31a8d4ec3577343f5645cdb1e3c WatchSource:0}: Error finding container b29190a22bb89d0cd73c0b4f5a09c20ea956d31a8d4ec3577343f5645cdb1e3c: Status 404 returned error can't find the container with id b29190a22bb89d0cd73c0b4f5a09c20ea956d31a8d4ec3577343f5645cdb1e3c Sep 11 12:15:50.350152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.350124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-rct9g"] Sep 11 12:15:50.427143 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.427106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rct9g" event={"ID":"1a3738d8-352f-430d-8336-91b2319e5f1f","Type":"ContainerStarted","Data":"b29190a22bb89d0cd73c0b4f5a09c20ea956d31a8d4ec3577343f5645cdb1e3c"} Sep 11 12:15:50.594104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.594076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.596516 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.596496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e848fd29-3e88-4e06-939d-2b3f37588c79-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-wvltp\" (UID: \"e848fd29-3e88-4e06-939d-2b3f37588c79\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:50.870525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:50.870493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:51.160565 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:51.160294 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp"] Sep 11 12:15:51.163056 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:15:51.163031 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode848fd29_3e88_4e06_939d_2b3f37588c79.slice/crio-f3ab7cf4c91174b74d6024e74997c77e20695e621abcde59425125b66da36ec6 WatchSource:0}: Error finding container f3ab7cf4c91174b74d6024e74997c77e20695e621abcde59425125b66da36ec6: Status 404 returned error can't find the container with id f3ab7cf4c91174b74d6024e74997c77e20695e621abcde59425125b66da36ec6 Sep 11 12:15:51.433018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:51.432926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" event={"ID":"e848fd29-3e88-4e06-939d-2b3f37588c79","Type":"ContainerStarted","Data":"f3ab7cf4c91174b74d6024e74997c77e20695e621abcde59425125b66da36ec6"} Sep 11 12:15:52.286213 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:52.286179 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:15:53.445019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:53.444937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" event={"ID":"e848fd29-3e88-4e06-939d-2b3f37588c79","Type":"ContainerStarted","Data":"12c189a06eff7bffca7bf328cd1c8927b42e9cc22cab0bf4e3169a00d65973cf"} Sep 11 12:15:53.445420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:53.445189 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:53.450830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:53.450806 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" Sep 11 12:15:53.465219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:53.465158 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-wvltp" podStartSLOduration=2.483984855 podStartE2EDuration="4.46513726s" totalImagesPullingTime="1.981152405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:15:53.464719456 +0000 UTC m=+92.952924514" watchObservedRunningTime="2026-09-11 12:15:53.46513726 +0000 UTC m=+92.953342319" Sep 11 12:15:58.293264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:15:58.293231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g6vhr" Sep 11 12:16:07.515866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.515771 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-rct9g" event={"ID":"1a3738d8-352f-430d-8336-91b2319e5f1f","Type":"ContainerStarted","Data":"02de1c58b2cb27c792f10ddfdc5a17d6096c06ca29447858cee879d08f25f5da"} Sep 11 12:16:07.516303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.515929 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:16:07.534440 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.534412 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-rct9g" Sep 11 12:16:07.564300 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.564246 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-rct9g" podStartSLOduration=1.703524206 podStartE2EDuration="18.564224562s" totalImagesPullingTime="16.860700356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:16:07.5390057 +0000 UTC m=+107.027210759" watchObservedRunningTime="2026-09-11 12:16:07.564224562 +0000 UTC m=+107.052429620" Sep 11 12:16:07.666015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.665979 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:16:07.669083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.669055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.672558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.672532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 12:16:07.674161 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.674139 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 12:16:07.674461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.674436 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 12:16:07.674888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.674864 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 12:16:07.675227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.675084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gvxtr\"" Sep 11 12:16:07.675419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.675398 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 12:16:07.679826 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.679802 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 12:16:07.683531 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.683513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:16:07.743764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.743730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.743936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.743783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.743936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.743815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.743936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.743921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.744105 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.743971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxkn\" (UniqueName: \"kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.744105 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.744014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.744196 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.744113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.844835 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.844757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.844835 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.844824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxkn\" (UniqueName: \"kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.844879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.844953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.844995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.845028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.845052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.845929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.845901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.846710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.846101 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.846710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.846549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.846710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.846672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.847974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.847950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.849757 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.849735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.858199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.858178 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxkn\" (UniqueName: \"kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn\") pod \"console-5cfdfcbd5c-rssm8\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:07.982343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:07.982303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:08.271487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:08.271452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:16:08.274116 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:16:08.274085 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49a850f_73a9_4ecc_8189_6912d68d23c7.slice/crio-c61702af59471a6feb106e99b66331dcc602e3438927b1c55a786a5f7440757b WatchSource:0}: Error finding container c61702af59471a6feb106e99b66331dcc602e3438927b1c55a786a5f7440757b: Status 404 returned error can't find the container with id c61702af59471a6feb106e99b66331dcc602e3438927b1c55a786a5f7440757b Sep 11 12:16:08.522814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:08.522718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdfcbd5c-rssm8" event={"ID":"a49a850f-73a9-4ecc-8189-6912d68d23c7","Type":"ContainerStarted","Data":"c61702af59471a6feb106e99b66331dcc602e3438927b1c55a786a5f7440757b"} Sep 11 12:16:13.550760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:13.550720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdfcbd5c-rssm8" event={"ID":"a49a850f-73a9-4ecc-8189-6912d68d23c7","Type":"ContainerStarted","Data":"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81"} Sep 11 12:16:13.572486 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:13.572436 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cfdfcbd5c-rssm8" podStartSLOduration=2.248026264 podStartE2EDuration="6.572421457s" totalImagesPullingTime="4.324395193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:16:13.571695547 +0000 UTC m=+113.059900608" watchObservedRunningTime="2026-09-11 12:16:13.572421457 +0000 UTC m=+113.060626515" Sep 11 12:16:17.313989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.313943 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" podUID="2e14ba71-3975-4253-a03a-aa566829d934" containerName="registry" containerID="cri-o://fb27028990654777a23a1c353ded1283d9f833939e87c494d3cb1a44b5a6053a" gracePeriod=30 Sep 11 12:16:17.570112 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.570050 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e14ba71-3975-4253-a03a-aa566829d934" containerID="fb27028990654777a23a1c353ded1283d9f833939e87c494d3cb1a44b5a6053a" exitCode=0 Sep 11 12:16:17.570219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.570113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" event={"ID":"2e14ba71-3975-4253-a03a-aa566829d934","Type":"ContainerDied","Data":"fb27028990654777a23a1c353ded1283d9f833939e87c494d3cb1a44b5a6053a"} Sep 11 12:16:17.570219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.570158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" event={"ID":"2e14ba71-3975-4253-a03a-aa566829d934","Type":"ContainerDied","Data":"a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d"} Sep 11 12:16:17.570219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.570179 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a400488d9e13c6aacae32c131a96010479cd4e213e2c0dcbadf92ef17addda3d" Sep 11 12:16:17.597464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.597444 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:16:17.733250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733223 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733255 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733274 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r64h\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733355 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733376 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733404 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733433 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets\") pod \"2e14ba71-3975-4253-a03a-aa566829d934\" (UID: \"2e14ba71-3975-4253-a03a-aa566829d934\") " Sep 11 12:16:17.733845 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733817 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:16:17.733975 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.733841 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:16:17.736077 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.736009 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:16:17.736188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.736157 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:16:17.736304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.736287 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h" (OuterVolumeSpecName: "kube-api-access-6r64h") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "kube-api-access-6r64h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:16:17.736485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.736467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:16:17.736554 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.736534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:16:17.742100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.742078 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2e14ba71-3975-4253-a03a-aa566829d934" (UID: "2e14ba71-3975-4253-a03a-aa566829d934"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:16:17.834203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834152 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-image-registry-private-configuration\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834174 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-bound-sa-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834186 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-registry-tls\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834194 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2e14ba71-3975-4253-a03a-aa566829d934-ca-trust-extracted\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834203 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2e14ba71-3975-4253-a03a-aa566829d934-installation-pull-secrets\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834394 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834212 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-registry-certificates\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834394 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834221 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2e14ba71-3975-4253-a03a-aa566829d934-trusted-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.834394 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.834229 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r64h\" (UniqueName: \"kubernetes.io/projected/2e14ba71-3975-4253-a03a-aa566829d934-kube-api-access-6r64h\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:16:17.982760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.982735 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:17.982847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.982798 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:17.987541 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:17.987523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:18.574625 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:18.574585 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ff64577df-s6xw8" Sep 11 12:16:18.578959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:18.578933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:16:18.617183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:18.617151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:16:18.620745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:18.620723 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ff64577df-s6xw8"] Sep 11 12:16:18.953394 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:18.953359 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e14ba71-3975-4253-a03a-aa566829d934" path="/var/lib/kubelet/pods/2e14ba71-3975-4253-a03a-aa566829d934/volumes" Sep 11 12:16:32.633027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.632987 2575 generic.go:356] "Generic (PLEG): container finished" podID="0874441a-169d-48eb-9424-330a62809713" containerID="664ad7b8aa3f5b2f93b36da8969bb761a8dfc95e61c4f51ab31f28a20d20f3f2" exitCode=0 Sep 11 12:16:32.633437 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.633066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s9tjz" event={"ID":"0874441a-169d-48eb-9424-330a62809713","Type":"ContainerDied","Data":"664ad7b8aa3f5b2f93b36da8969bb761a8dfc95e61c4f51ab31f28a20d20f3f2"} Sep 11 12:16:32.633527 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.633507 2575 scope.go:117] "RemoveContainer" containerID="664ad7b8aa3f5b2f93b36da8969bb761a8dfc95e61c4f51ab31f28a20d20f3f2" Sep 11 12:16:32.635161 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.635097 2575 generic.go:356] "Generic (PLEG): container finished" podID="b889fdfc-8f57-4240-abd7-1a1ed43366aa" containerID="0cd001ef71e4df0bd33251af3b9ca9e83ef800eeb4c6e57c603509b96c4db217" exitCode=0 Sep 11 12:16:32.635236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.635169 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" event={"ID":"b889fdfc-8f57-4240-abd7-1a1ed43366aa","Type":"ContainerDied","Data":"0cd001ef71e4df0bd33251af3b9ca9e83ef800eeb4c6e57c603509b96c4db217"} Sep 11 12:16:32.635526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:32.635508 2575 scope.go:117] "RemoveContainer" containerID="0cd001ef71e4df0bd33251af3b9ca9e83ef800eeb4c6e57c603509b96c4db217" Sep 11 12:16:33.640795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:33.640759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s9tjz" event={"ID":"0874441a-169d-48eb-9424-330a62809713","Type":"ContainerStarted","Data":"17d6792520daace498360f6cea8ded3a09c997e5fb14d3720f7aa808a5c96fc3"} Sep 11 12:16:33.642718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:33.642693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qrt64" event={"ID":"b889fdfc-8f57-4240-abd7-1a1ed43366aa","Type":"ContainerStarted","Data":"91ee9c3c2e9ae7eb1e650b1414f7e5f6b5fea500292d3e2c920968c0d3de39a2"} Sep 11 12:16:43.684074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:43.684036 2575 generic.go:356] "Generic (PLEG): container finished" podID="8da8d03e-56f4-4451-b35a-4e4a4dfe6f72" containerID="9c047e4fbe56b1308f4d1042fe6b2161e4c9b809695c6f007626ec5a16400a00" exitCode=0 Sep 11 12:16:43.684513 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:43.684112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" event={"ID":"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72","Type":"ContainerDied","Data":"9c047e4fbe56b1308f4d1042fe6b2161e4c9b809695c6f007626ec5a16400a00"} Sep 11 12:16:43.684560 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:43.684536 2575 scope.go:117] "RemoveContainer" containerID="9c047e4fbe56b1308f4d1042fe6b2161e4c9b809695c6f007626ec5a16400a00" Sep 11 12:16:44.691090 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:16:44.691061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-tjpvn" event={"ID":"8da8d03e-56f4-4451-b35a-4e4a4dfe6f72","Type":"ContainerStarted","Data":"ed98e262a8636a4c25bc5b73a8461428bc83c3c03b028bc6d09adde2bde6efaf"} Sep 11 12:17:14.763824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.763741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:17:14.764315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.764093 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e14ba71-3975-4253-a03a-aa566829d934" containerName="registry" Sep 11 12:17:14.764315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.764107 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e14ba71-3975-4253-a03a-aa566829d934" containerName="registry" Sep 11 12:17:14.764315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.764175 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e14ba71-3975-4253-a03a-aa566829d934" containerName="registry" Sep 11 12:17:14.766018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.765992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.790352 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.790327 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:17:14.836995 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.836960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwr5f\" (UniqueName: \"kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.837304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.837171 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938273 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938476 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938548 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwr5f\" (UniqueName: \"kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.938748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.938631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.939298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.939219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.939298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.939244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.939298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.939273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.939636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.939587 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.941046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.941020 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.941175 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.941153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:14.953698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:14.953673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwr5f\" (UniqueName: \"kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f\") pod \"console-85b98d4fdf-sqbtw\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:15.075622 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:15.075538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:15.367261 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:15.367192 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:17:15.829726 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:15.829691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b98d4fdf-sqbtw" event={"ID":"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f","Type":"ContainerStarted","Data":"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306"} Sep 11 12:17:15.830134 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:15.829744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b98d4fdf-sqbtw" event={"ID":"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f","Type":"ContainerStarted","Data":"79f750d7322986421dc6bec1a03c15905c8b68821381c8eedad0125bfb8962ae"} Sep 11 12:17:15.856438 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:15.856390 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85b98d4fdf-sqbtw" podStartSLOduration=1.8563737869999999 podStartE2EDuration="1.856373787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:17:15.856226128 +0000 UTC m=+175.344431187" watchObservedRunningTime="2026-09-11 12:17:15.856373787 +0000 UTC m=+175.344578843" Sep 11 12:17:25.076500 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:25.076461 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:25.077011 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:25.076519 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:25.081371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:25.081344 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:25.879240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:25.879209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:17:25.943543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:25.943507 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:17:50.967981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:50.967910 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cfdfcbd5c-rssm8" podUID="a49a850f-73a9-4ecc-8189-6912d68d23c7" containerName="console" containerID="cri-o://f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81" gracePeriod=15 Sep 11 12:17:51.209008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.208984 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:17:51.302834 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.302778 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.302834 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.302821 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.302965 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.302943 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.303001 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.302976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.303059 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxkn\" (UniqueName: \"kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.303111 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303068 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.303111 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert\") pod \"a49a850f-73a9-4ecc-8189-6912d68d23c7\" (UID: \"a49a850f-73a9-4ecc-8189-6912d68d23c7\") " Sep 11 12:17:51.303390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303354 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config" (OuterVolumeSpecName: "console-config") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:17:51.303487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303407 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:17:51.303643 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303600 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:17:51.303756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.303728 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca" (OuterVolumeSpecName: "service-ca") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:17:51.305180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.305162 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:17:51.305651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.305624 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn" (OuterVolumeSpecName: "kube-api-access-5dxkn") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "kube-api-access-5dxkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:17:51.305710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.305643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a49a850f-73a9-4ecc-8189-6912d68d23c7" (UID: "a49a850f-73a9-4ecc-8189-6912d68d23c7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:17:51.404320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404305 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-trusted-ca-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404322 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404332 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxkn\" (UniqueName: \"kubernetes.io/projected/a49a850f-73a9-4ecc-8189-6912d68d23c7-kube-api-access-5dxkn\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404341 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404350 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a49a850f-73a9-4ecc-8189-6912d68d23c7-oauth-serving-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404359 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-serving-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.404413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.404367 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a49a850f-73a9-4ecc-8189-6912d68d23c7-console-oauth-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:17:51.984213 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.984181 2575 generic.go:356] "Generic (PLEG): container finished" podID="a49a850f-73a9-4ecc-8189-6912d68d23c7" containerID="f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81" exitCode=0 Sep 11 12:17:51.984589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.984232 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfdfcbd5c-rssm8" Sep 11 12:17:51.984589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.984269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdfcbd5c-rssm8" event={"ID":"a49a850f-73a9-4ecc-8189-6912d68d23c7","Type":"ContainerDied","Data":"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81"} Sep 11 12:17:51.984589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.984326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfdfcbd5c-rssm8" event={"ID":"a49a850f-73a9-4ecc-8189-6912d68d23c7","Type":"ContainerDied","Data":"c61702af59471a6feb106e99b66331dcc602e3438927b1c55a786a5f7440757b"} Sep 11 12:17:51.984589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.984351 2575 scope.go:117] "RemoveContainer" containerID="f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81" Sep 11 12:17:51.999804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:51.999782 2575 scope.go:117] "RemoveContainer" containerID="f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81" Sep 11 12:17:52.000125 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:17:52.000103 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81\": container with ID starting with f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81 not found: ID does not exist" containerID="f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81" Sep 11 12:17:52.000246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:52.000130 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81"} err="failed to get container status \"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81\": rpc error: code = NotFound desc = could not find container \"f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81\": container with ID starting with f04077255f4692f71fc468463c2b9787ed92b801690c328347bcd91fb608da81 not found: ID does not exist" Sep 11 12:17:52.011542 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:52.011516 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:17:52.019371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:52.019348 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cfdfcbd5c-rssm8"] Sep 11 12:17:52.953403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:17:52.953362 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49a850f-73a9-4ecc-8189-6912d68d23c7" path="/var/lib/kubelet/pods/a49a850f-73a9-4ecc-8189-6912d68d23c7/volumes" Sep 11 12:18:45.005271 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:18:45.005182 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:19:10.026992 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.026942 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85b98d4fdf-sqbtw" podUID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" containerName="console" containerID="cri-o://09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306" gracePeriod=15 Sep 11 12:19:10.263279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.263255 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:19:10.312131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.312041 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" containerID="09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306" exitCode=0 Sep 11 12:19:10.312131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.312092 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85b98d4fdf-sqbtw" Sep 11 12:19:10.312296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.312133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b98d4fdf-sqbtw" event={"ID":"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f","Type":"ContainerDied","Data":"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306"} Sep 11 12:19:10.312296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.312184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85b98d4fdf-sqbtw" event={"ID":"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f","Type":"ContainerDied","Data":"79f750d7322986421dc6bec1a03c15905c8b68821381c8eedad0125bfb8962ae"} Sep 11 12:19:10.312296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.312207 2575 scope.go:117] "RemoveContainer" containerID="09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306" Sep 11 12:19:10.320512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.320492 2575 scope.go:117] "RemoveContainer" containerID="09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306" Sep 11 12:19:10.320808 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:19:10.320788 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306\": container with ID starting with 09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306 not found: ID does not exist" containerID="09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306" Sep 11 12:19:10.320875 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.320818 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306"} err="failed to get container status \"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306\": rpc error: code = NotFound desc = could not find container \"09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306\": container with ID starting with 09ef175655ebd7514decdfa9fd0db2439dc95f6d03538f1cd4cb465862c31306 not found: ID does not exist" Sep 11 12:19:10.361352 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361328 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361376 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361424 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361490 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwr5f\" (UniqueName: \"kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361515 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle\") pod \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\" (UID: \"1b4e1db7-03a7-4a15-afc7-9dbb6a59097f\") " Sep 11 12:19:10.361829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:19:10.361829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361811 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config" (OuterVolumeSpecName: "console-config") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:19:10.361988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.361962 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:19:10.362042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.362023 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:19:10.363783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.363758 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:19:10.364120 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.364097 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:19:10.364203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.364112 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f" (OuterVolumeSpecName: "kube-api-access-mwr5f") pod "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" (UID: "1b4e1db7-03a7-4a15-afc7-9dbb6a59097f"). InnerVolumeSpecName "kube-api-access-mwr5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:19:10.462650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462587 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-serving-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462645 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462656 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-oauth-serving-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462665 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-oauth-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462673 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwr5f\" (UniqueName: \"kubernetes.io/projected/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-kube-api-access-mwr5f\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462682 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-trusted-ca-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.462880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.462690 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f-console-config\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:10.636342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.636312 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:19:10.639700 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.639679 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85b98d4fdf-sqbtw"] Sep 11 12:19:10.952142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:10.952114 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" path="/var/lib/kubelet/pods/1b4e1db7-03a7-4a15-afc7-9dbb6a59097f/volumes" Sep 11 12:19:20.860025 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:20.859998 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:19:20.864415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:20.864391 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:19:20.873295 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:20.873272 2575 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 12:19:33.673991 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.673951 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7"] Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674259 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" containerName="console" Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674271 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" containerName="console" Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674296 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49a850f-73a9-4ecc-8189-6912d68d23c7" containerName="console" Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674301 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49a850f-73a9-4ecc-8189-6912d68d23c7" containerName="console" Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674369 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b4e1db7-03a7-4a15-afc7-9dbb6a59097f" containerName="console" Sep 11 12:19:33.675347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.674381 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49a850f-73a9-4ecc-8189-6912d68d23c7" containerName="console" Sep 11 12:19:33.676266 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.676226 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.678911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.678887 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 12:19:33.679959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.679942 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4m7h\"" Sep 11 12:19:33.680255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.680235 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 12:19:33.686578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.686549 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7"] Sep 11 12:19:33.827561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.827535 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.827704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.827566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2mkq\" (UniqueName: \"kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.827704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.827597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.928575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.928514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.928575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.928545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2mkq\" (UniqueName: \"kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.928575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.928575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.928948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.928923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.928998 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.928954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.938347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.938318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2mkq\" (UniqueName: \"kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:33.986827 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:33.986804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:34.260071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:34.259999 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7"] Sep 11 12:19:34.262968 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:19:34.262931 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode501fd41_750d_4f0e_a9b4_a03be8a9db97.slice/crio-e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5 WatchSource:0}: Error finding container e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5: Status 404 returned error can't find the container with id e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5 Sep 11 12:19:34.265227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:34.265208 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:19:34.414886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:34.414853 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" event={"ID":"e501fd41-750d-4f0e-a9b4-a03be8a9db97","Type":"ContainerStarted","Data":"e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5"} Sep 11 12:19:42.450977 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:42.450941 2575 generic.go:356] "Generic (PLEG): container finished" podID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerID="6722e6a6cd31d97b477aa26af7266b1d94bae1cc2fd41d09008a6803a9f3ebff" exitCode=0 Sep 11 12:19:42.451382 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:42.451033 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" event={"ID":"e501fd41-750d-4f0e-a9b4-a03be8a9db97","Type":"ContainerDied","Data":"6722e6a6cd31d97b477aa26af7266b1d94bae1cc2fd41d09008a6803a9f3ebff"} Sep 11 12:19:45.466307 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:45.466268 2575 generic.go:356] "Generic (PLEG): container finished" podID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerID="7faf30f375b5d99b0f015d6148fa1c87963b65818bdaf3093cf5a3a3421b0622" exitCode=0 Sep 11 12:19:45.466703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:45.466345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" event={"ID":"e501fd41-750d-4f0e-a9b4-a03be8a9db97","Type":"ContainerDied","Data":"7faf30f375b5d99b0f015d6148fa1c87963b65818bdaf3093cf5a3a3421b0622"} Sep 11 12:19:53.503590 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:53.503562 2575 generic.go:356] "Generic (PLEG): container finished" podID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerID="18a58aab7d032c353f099c96cb39ae134c0a3b62ec8c9543adb3ba0bcae7a52e" exitCode=0 Sep 11 12:19:53.503989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:53.503658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" event={"ID":"e501fd41-750d-4f0e-a9b4-a03be8a9db97","Type":"ContainerDied","Data":"18a58aab7d032c353f099c96cb39ae134c0a3b62ec8c9543adb3ba0bcae7a52e"} Sep 11 12:19:54.635476 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.635449 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:54.785254 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.785189 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle\") pod \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " Sep 11 12:19:54.785254 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.785223 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2mkq\" (UniqueName: \"kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq\") pod \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " Sep 11 12:19:54.785428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.785268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util\") pod \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\" (UID: \"e501fd41-750d-4f0e-a9b4-a03be8a9db97\") " Sep 11 12:19:54.785884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.785848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle" (OuterVolumeSpecName: "bundle") pod "e501fd41-750d-4f0e-a9b4-a03be8a9db97" (UID: "e501fd41-750d-4f0e-a9b4-a03be8a9db97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:19:54.787591 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.787555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq" (OuterVolumeSpecName: "kube-api-access-z2mkq") pod "e501fd41-750d-4f0e-a9b4-a03be8a9db97" (UID: "e501fd41-750d-4f0e-a9b4-a03be8a9db97"). InnerVolumeSpecName "kube-api-access-z2mkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:19:54.789435 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.789415 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util" (OuterVolumeSpecName: "util") pod "e501fd41-750d-4f0e-a9b4-a03be8a9db97" (UID: "e501fd41-750d-4f0e-a9b4-a03be8a9db97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:19:54.886151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.886117 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:54.886151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.886149 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2mkq\" (UniqueName: \"kubernetes.io/projected/e501fd41-750d-4f0e-a9b4-a03be8a9db97-kube-api-access-z2mkq\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:54.886151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:54.886157 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e501fd41-750d-4f0e-a9b4-a03be8a9db97-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:19:55.515253 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:55.515224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" event={"ID":"e501fd41-750d-4f0e-a9b4-a03be8a9db97","Type":"ContainerDied","Data":"e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5"} Sep 11 12:19:55.515371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:55.515256 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402txh7" Sep 11 12:19:55.515418 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:19:55.515276 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e057df24db48bc5c1717b12f1befc2b5149be1317b07b2de8e2abd6bf94470f5" Sep 11 12:20:01.749278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749243 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n"] Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749540 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="util" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749551 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="util" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749569 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="pull" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749575 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="pull" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749583 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="extract" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749589 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="extract" Sep 11 12:20:01.749710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.749668 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e501fd41-750d-4f0e-a9b4-a03be8a9db97" containerName="extract" Sep 11 12:20:01.754313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.754295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:01.761683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.761662 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 12:20:01.761983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.761965 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b8tkm\"" Sep 11 12:20:01.762330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.762315 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 12:20:01.776171 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.776147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n"] Sep 11 12:20:01.933595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.933567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmct\" (UniqueName: \"kubernetes.io/projected/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-kube-api-access-wcmct\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:01.933718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:01.933672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.034314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.034257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.034406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.034317 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmct\" (UniqueName: \"kubernetes.io/projected/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-kube-api-access-wcmct\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.034634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.034590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.044574 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.044554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmct\" (UniqueName: \"kubernetes.io/projected/795ffa45-c7ad-4f79-8910-2d02bd4dc7fc-kube-api-access-wcmct\") pod \"cert-manager-operator-controller-manager-55c444bf86-9vc9n\" (UID: \"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.063014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.062995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" Sep 11 12:20:02.342591 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.342567 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n"] Sep 11 12:20:02.344549 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:02.344517 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795ffa45_c7ad_4f79_8910_2d02bd4dc7fc.slice/crio-c279558f03bd2a7dfd3b539e18813e6c1dc3d2780a7b5d173c8a6a4be4642d0d WatchSource:0}: Error finding container c279558f03bd2a7dfd3b539e18813e6c1dc3d2780a7b5d173c8a6a4be4642d0d: Status 404 returned error can't find the container with id c279558f03bd2a7dfd3b539e18813e6c1dc3d2780a7b5d173c8a6a4be4642d0d Sep 11 12:20:02.544235 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:02.544206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" event={"ID":"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc","Type":"ContainerStarted","Data":"c279558f03bd2a7dfd3b539e18813e6c1dc3d2780a7b5d173c8a6a4be4642d0d"} Sep 11 12:20:04.556062 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:04.556017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" event={"ID":"795ffa45-c7ad-4f79-8910-2d02bd4dc7fc","Type":"ContainerStarted","Data":"acc78ffbd5764f13dccc0480e7367d93ef4e82a45d89942c5491b6845cbcff1e"} Sep 11 12:20:04.582855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:04.582801 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9vc9n" podStartSLOduration=1.8318963940000002 podStartE2EDuration="3.582782848s" totalImagesPullingTime="1.750886454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:04.580353846 +0000 UTC m=+344.068558894" watchObservedRunningTime="2026-09-11 12:20:04.582782848 +0000 UTC m=+344.070987907" Sep 11 12:20:05.604307 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.604279 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft"] Sep 11 12:20:05.606795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.606767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.609740 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.609719 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4m7h\"" Sep 11 12:20:05.609853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.609719 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 12:20:05.610543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.610517 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 12:20:05.631089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.631068 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft"] Sep 11 12:20:05.662938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.662915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.663040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.662974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrk5\" (UniqueName: \"kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.663084 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.663035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.763580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.763554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrk5\" (UniqueName: \"kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.763713 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.763587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.763713 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.763659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.764018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.764001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.764065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.764018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.772752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.772731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrk5\" (UniqueName: \"kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:05.921707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:05.921683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:06.194583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:06.194516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft"] Sep 11 12:20:06.198036 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:06.198006 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7751d98_6bd3_4a5d_99c1_4f4a739e8427.slice/crio-4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800 WatchSource:0}: Error finding container 4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800: Status 404 returned error can't find the container with id 4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800 Sep 11 12:20:06.567302 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:06.567233 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerID="ac938560950c2015b478b96166745b74ca2a918688138bcc91f0b8721acbad4c" exitCode=0 Sep 11 12:20:06.567490 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:06.567322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" event={"ID":"d7751d98-6bd3-4a5d-99c1-4f4a739e8427","Type":"ContainerDied","Data":"ac938560950c2015b478b96166745b74ca2a918688138bcc91f0b8721acbad4c"} Sep 11 12:20:06.567490 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:06.567400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" event={"ID":"d7751d98-6bd3-4a5d-99c1-4f4a739e8427","Type":"ContainerStarted","Data":"4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800"} Sep 11 12:20:07.524827 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.524791 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dgpxc"] Sep 11 12:20:07.526892 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.526873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.529905 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.529877 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ctbzj\"" Sep 11 12:20:07.530056 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.529880 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 12:20:07.530056 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.529880 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 12:20:07.542642 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.542593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dgpxc"] Sep 11 12:20:07.576257 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.576237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2mf\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-kube-api-access-4p2mf\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.576257 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.576266 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.677571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.677545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2mf\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-kube-api-access-4p2mf\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.677683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.677576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.687703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.687672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.687808 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.687784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2mf\" (UniqueName: \"kubernetes.io/projected/ae236e8a-32f9-4123-97ac-2280cdfa0e29-kube-api-access-4p2mf\") pod \"cert-manager-webhook-6486ffd997-dgpxc\" (UID: \"ae236e8a-32f9-4123-97ac-2280cdfa0e29\") " pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:07.842791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:07.842737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:08.128412 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:08.128327 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-dgpxc"] Sep 11 12:20:08.131208 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:08.131175 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae236e8a_32f9_4123_97ac_2280cdfa0e29.slice/crio-7bc9cefeefc5bc030d563b121c36cf2c39e855129da58d84e21033ec40fcbae6 WatchSource:0}: Error finding container 7bc9cefeefc5bc030d563b121c36cf2c39e855129da58d84e21033ec40fcbae6: Status 404 returned error can't find the container with id 7bc9cefeefc5bc030d563b121c36cf2c39e855129da58d84e21033ec40fcbae6 Sep 11 12:20:08.577674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:08.577640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" event={"ID":"ae236e8a-32f9-4123-97ac-2280cdfa0e29","Type":"ContainerStarted","Data":"7bc9cefeefc5bc030d563b121c36cf2c39e855129da58d84e21033ec40fcbae6"} Sep 11 12:20:09.586863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:09.586780 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerID="d79613b32a2c8d4383ad5025c6349b72e912026305a68937eeb1dc06f4b0a986" exitCode=0 Sep 11 12:20:09.586863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:09.586821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" event={"ID":"d7751d98-6bd3-4a5d-99c1-4f4a739e8427","Type":"ContainerDied","Data":"d79613b32a2c8d4383ad5025c6349b72e912026305a68937eeb1dc06f4b0a986"} Sep 11 12:20:10.594888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:10.594853 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerID="624e56659abcdecd04aea5ce6ce7b087121daf602c5ff74efe4de82ff2cdab20" exitCode=0 Sep 11 12:20:10.595356 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:10.594917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" event={"ID":"d7751d98-6bd3-4a5d-99c1-4f4a739e8427","Type":"ContainerDied","Data":"624e56659abcdecd04aea5ce6ce7b087121daf602c5ff74efe4de82ff2cdab20"} Sep 11 12:20:11.600786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.600702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" event={"ID":"ae236e8a-32f9-4123-97ac-2280cdfa0e29","Type":"ContainerStarted","Data":"d8ade16f89a9a3fd5357767819c6fbae2c1ab18ccfc2097bd7a4de43bae3b08d"} Sep 11 12:20:11.600786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.600762 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:11.619851 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.619798 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" podStartSLOduration=1.52380591 podStartE2EDuration="4.619777452s" totalImagesPullingTime="3.095971542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:11.618492704 +0000 UTC m=+351.106697762" watchObservedRunningTime="2026-09-11 12:20:11.619777452 +0000 UTC m=+351.107982510" Sep 11 12:20:11.726355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.726334 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:11.811433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.811406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle\") pod \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " Sep 11 12:20:11.811604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.811586 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util\") pod \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " Sep 11 12:20:11.811696 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.811648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljrk5\" (UniqueName: \"kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5\") pod \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\" (UID: \"d7751d98-6bd3-4a5d-99c1-4f4a739e8427\") " Sep 11 12:20:11.811755 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.811737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle" (OuterVolumeSpecName: "bundle") pod "d7751d98-6bd3-4a5d-99c1-4f4a739e8427" (UID: "d7751d98-6bd3-4a5d-99c1-4f4a739e8427"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:11.811977 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.811963 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:11.813719 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.813703 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5" (OuterVolumeSpecName: "kube-api-access-ljrk5") pod "d7751d98-6bd3-4a5d-99c1-4f4a739e8427" (UID: "d7751d98-6bd3-4a5d-99c1-4f4a739e8427"). InnerVolumeSpecName "kube-api-access-ljrk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:20:11.815502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.815478 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util" (OuterVolumeSpecName: "util") pod "d7751d98-6bd3-4a5d-99c1-4f4a739e8427" (UID: "d7751d98-6bd3-4a5d-99c1-4f4a739e8427"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:11.912267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.912246 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:11.912267 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:11.912265 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrk5\" (UniqueName: \"kubernetes.io/projected/d7751d98-6bd3-4a5d-99c1-4f4a739e8427-kube-api-access-ljrk5\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:12.608689 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:12.608660 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" Sep 11 12:20:12.608689 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:12.608674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvlvft" event={"ID":"d7751d98-6bd3-4a5d-99c1-4f4a739e8427","Type":"ContainerDied","Data":"4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800"} Sep 11 12:20:12.609198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:12.608727 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ce9fb0619bdb86218bb05b18525ecff83ef05efd1b8185e01533f925cd22800" Sep 11 12:20:16.402094 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402050 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lhkwb"] Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402397 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="util" Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402415 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="util" Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402428 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="extract" Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402434 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="extract" Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402450 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="pull" Sep 11 12:20:16.402503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402455 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="pull" Sep 11 12:20:16.402727 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.402538 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7751d98-6bd3-4a5d-99c1-4f4a739e8427" containerName="extract" Sep 11 12:20:16.405653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.405605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.408445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.408418 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wlkkm\"" Sep 11 12:20:16.415730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.415699 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lhkwb"] Sep 11 12:20:16.442876 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.442847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6nj\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-kube-api-access-4v6nj\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.442971 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.442882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-bound-sa-token\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.543857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.543830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6nj\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-kube-api-access-4v6nj\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.543967 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.543865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-bound-sa-token\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.552177 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.552147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-bound-sa-token\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.552357 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.552338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6nj\" (UniqueName: \"kubernetes.io/projected/eb0672df-4485-460c-8fae-999a64ea9290-kube-api-access-4v6nj\") pod \"cert-manager-55b4d954d5-lhkwb\" (UID: \"eb0672df-4485-460c-8fae-999a64ea9290\") " pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.717112 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.717059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lhkwb" Sep 11 12:20:16.992602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:16.992529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lhkwb"] Sep 11 12:20:16.995317 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:16.995289 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb0672df_4485_460c_8fae_999a64ea9290.slice/crio-f79ba97a01f302a2067b4f1c0b902174304147b06ec487f5a8a68979d270c850 WatchSource:0}: Error finding container f79ba97a01f302a2067b4f1c0b902174304147b06ec487f5a8a68979d270c850: Status 404 returned error can't find the container with id f79ba97a01f302a2067b4f1c0b902174304147b06ec487f5a8a68979d270c850 Sep 11 12:20:17.611015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:17.610987 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-dgpxc" Sep 11 12:20:17.635222 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:17.635150 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lhkwb" event={"ID":"eb0672df-4485-460c-8fae-999a64ea9290","Type":"ContainerStarted","Data":"5c90d4a81baf819f07c201ac9b6bfc27bda64695ecedc88f894fd00041924e67"} Sep 11 12:20:17.635362 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:17.635268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lhkwb" event={"ID":"eb0672df-4485-460c-8fae-999a64ea9290","Type":"ContainerStarted","Data":"f79ba97a01f302a2067b4f1c0b902174304147b06ec487f5a8a68979d270c850"} Sep 11 12:20:17.653043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:17.652983 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lhkwb" podStartSLOduration=1.652962925 podStartE2EDuration="1.652962925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:17.651415686 +0000 UTC m=+357.139620749" watchObservedRunningTime="2026-09-11 12:20:17.652962925 +0000 UTC m=+357.141167983" Sep 11 12:20:25.750589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.750560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts"] Sep 11 12:20:25.752821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.752806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.755480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.755457 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 12:20:25.756498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.756475 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 12:20:25.756498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.756492 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4m7h\"" Sep 11 12:20:25.763596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.763575 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts"] Sep 11 12:20:25.811070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.811045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxpw\" (UniqueName: \"kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.811160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.811076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.811160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.811112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.911945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.911921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.912044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.911963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxpw\" (UniqueName: \"kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.912044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.911983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.912268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.912252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.912325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.912279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:25.921060 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:25.921039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxpw\" (UniqueName: \"kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:26.061947 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:26.061901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:26.338802 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:26.338760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts"] Sep 11 12:20:26.341156 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:26.341129 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d32073a_25ad_4e51_8ca2_8d6f2dd04874.slice/crio-5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797 WatchSource:0}: Error finding container 5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797: Status 404 returned error can't find the container with id 5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797 Sep 11 12:20:26.673530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:26.673494 2575 generic.go:356] "Generic (PLEG): container finished" podID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerID="61ef78ca649a5826ae8215854ddc280c246a49cabd28ee07ceebad7ff26b7e2a" exitCode=0 Sep 11 12:20:26.673666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:26.673581 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" event={"ID":"0d32073a-25ad-4e51-8ca2-8d6f2dd04874","Type":"ContainerDied","Data":"61ef78ca649a5826ae8215854ddc280c246a49cabd28ee07ceebad7ff26b7e2a"} Sep 11 12:20:26.673666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:26.673652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" event={"ID":"0d32073a-25ad-4e51-8ca2-8d6f2dd04874","Type":"ContainerStarted","Data":"5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797"} Sep 11 12:20:27.679323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:27.679299 2575 generic.go:356] "Generic (PLEG): container finished" podID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerID="07d4db522ccf8ce437df0aa3f6fe9a36840a9e34a6c0ee8222577e279e3da79b" exitCode=0 Sep 11 12:20:27.679587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:27.679339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" event={"ID":"0d32073a-25ad-4e51-8ca2-8d6f2dd04874","Type":"ContainerDied","Data":"07d4db522ccf8ce437df0aa3f6fe9a36840a9e34a6c0ee8222577e279e3da79b"} Sep 11 12:20:28.685628 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:28.685585 2575 generic.go:356] "Generic (PLEG): container finished" podID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerID="193f3e3518bf74e55b5c72d1585ea354c8fa0fde80da1f9872bca88f51ca11d3" exitCode=0 Sep 11 12:20:28.685947 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:28.685639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" event={"ID":"0d32073a-25ad-4e51-8ca2-8d6f2dd04874","Type":"ContainerDied","Data":"193f3e3518bf74e55b5c72d1585ea354c8fa0fde80da1f9872bca88f51ca11d3"} Sep 11 12:20:29.819725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.819702 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:29.944469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.944439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util\") pod \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " Sep 11 12:20:29.944469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.944471 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle\") pod \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " Sep 11 12:20:29.944695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.944516 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhxpw\" (UniqueName: \"kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw\") pod \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\" (UID: \"0d32073a-25ad-4e51-8ca2-8d6f2dd04874\") " Sep 11 12:20:29.945233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.945209 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle" (OuterVolumeSpecName: "bundle") pod "0d32073a-25ad-4e51-8ca2-8d6f2dd04874" (UID: "0d32073a-25ad-4e51-8ca2-8d6f2dd04874"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:29.946666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.946641 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw" (OuterVolumeSpecName: "kube-api-access-vhxpw") pod "0d32073a-25ad-4e51-8ca2-8d6f2dd04874" (UID: "0d32073a-25ad-4e51-8ca2-8d6f2dd04874"). InnerVolumeSpecName "kube-api-access-vhxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:20:29.949962 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:29.949926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util" (OuterVolumeSpecName: "util") pod "0d32073a-25ad-4e51-8ca2-8d6f2dd04874" (UID: "0d32073a-25ad-4e51-8ca2-8d6f2dd04874"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:30.045424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.045401 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:30.045424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.045421 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:30.045538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.045430 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxpw\" (UniqueName: \"kubernetes.io/projected/0d32073a-25ad-4e51-8ca2-8d6f2dd04874-kube-api-access-vhxpw\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:30.696770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.696744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" event={"ID":"0d32073a-25ad-4e51-8ca2-8d6f2dd04874","Type":"ContainerDied","Data":"5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797"} Sep 11 12:20:30.696770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.696768 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afk78ts" Sep 11 12:20:30.696876 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:30.696789 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f1705e4ed04214a6fa175e1e6a442fde87238223838de021443f08c6cb6e797" Sep 11 12:20:39.978202 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978165 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q"] Sep 11 12:20:39.978662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978584 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="util" Sep 11 12:20:39.978662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978602 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="util" Sep 11 12:20:39.978662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978640 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="pull" Sep 11 12:20:39.978662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978648 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="pull" Sep 11 12:20:39.978880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978674 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="extract" Sep 11 12:20:39.978880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978682 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="extract" Sep 11 12:20:39.978880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.978763 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d32073a-25ad-4e51-8ca2-8d6f2dd04874" containerName="extract" Sep 11 12:20:39.983974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.983951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:39.986234 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.986211 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 12:20:39.986483 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.986469 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 12:20:39.987251 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.987238 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4m7h\"" Sep 11 12:20:39.993086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.993066 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw"] Sep 11 12:20:39.995959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.995941 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:39.996943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.996919 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q"] Sep 11 12:20:39.998516 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.998494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 12:20:39.998627 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.998507 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 12:20:39.998708 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.998692 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 12:20:39.998777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.998758 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 12:20:39.998969 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:39.998946 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n5jwr\"" Sep 11 12:20:40.072008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.071986 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw"] Sep 11 12:20:40.113069 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.113163 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.113163 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz9l\" (UniqueName: \"kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.113163 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.113277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.113277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.113277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.113277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.113257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5kth\" (UniqueName: \"kubernetes.io/projected/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-kube-api-access-p5kth\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.213821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.213912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.213912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.213912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.213912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5kth\" (UniqueName: \"kubernetes.io/projected/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-kube-api-access-p5kth\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.214074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.214074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.213987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.214074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.214013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz9l\" (UniqueName: \"kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.214265 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.214243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.214368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.214261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.214422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.214400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-manifests\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.216296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.216279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-webhook-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.216373 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.216296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-tmp\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.216429 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.216414 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.223488 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.223469 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz9l\" (UniqueName: \"kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.223668 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.223650 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5kth\" (UniqueName: \"kubernetes.io/projected/5bac7e8a-3405-4eea-abc3-2b2f5b3685af-kube-api-access-p5kth\") pod \"opendatahub-operator-controller-manager-8dd64745b-z59lw\" (UID: \"5bac7e8a-3405-4eea-abc3-2b2f5b3685af\") " pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.293598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.293551 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:40.306246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.306222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:40.570275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.570215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q"] Sep 11 12:20:40.572946 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:40.572916 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc9eaef_2205_4b44_926f_cb7799674915.slice/crio-d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1 WatchSource:0}: Error finding container d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1: Status 404 returned error can't find the container with id d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1 Sep 11 12:20:40.590368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.590334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw"] Sep 11 12:20:40.595831 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:20:40.595793 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bac7e8a_3405_4eea_abc3_2b2f5b3685af.slice/crio-75302003f29a3eeef2f2b5d9eb3273aefef5b9f99f5342645f2d98045ecc58bb WatchSource:0}: Error finding container 75302003f29a3eeef2f2b5d9eb3273aefef5b9f99f5342645f2d98045ecc58bb: Status 404 returned error can't find the container with id 75302003f29a3eeef2f2b5d9eb3273aefef5b9f99f5342645f2d98045ecc58bb Sep 11 12:20:40.739587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.739525 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cc9eaef-2205-4b44-926f-cb7799674915" containerID="cd3e0c393211f2b1aea3d501e8ae2a38edf794379dc719a779afee87702e3182" exitCode=0 Sep 11 12:20:40.739698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.739629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" event={"ID":"6cc9eaef-2205-4b44-926f-cb7799674915","Type":"ContainerDied","Data":"cd3e0c393211f2b1aea3d501e8ae2a38edf794379dc719a779afee87702e3182"} Sep 11 12:20:40.739748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.739710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" event={"ID":"6cc9eaef-2205-4b44-926f-cb7799674915","Type":"ContainerStarted","Data":"d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1"} Sep 11 12:20:40.741148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:40.741127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" event={"ID":"5bac7e8a-3405-4eea-abc3-2b2f5b3685af","Type":"ContainerStarted","Data":"75302003f29a3eeef2f2b5d9eb3273aefef5b9f99f5342645f2d98045ecc58bb"} Sep 11 12:20:41.748818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:41.748745 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cc9eaef-2205-4b44-926f-cb7799674915" containerID="a996e841b57c98783f6458ec714fd0cb10088675faff470c61ae3cf18144ec94" exitCode=0 Sep 11 12:20:41.749193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:41.748834 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" event={"ID":"6cc9eaef-2205-4b44-926f-cb7799674915","Type":"ContainerDied","Data":"a996e841b57c98783f6458ec714fd0cb10088675faff470c61ae3cf18144ec94"} Sep 11 12:20:42.756660 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:42.756598 2575 generic.go:356] "Generic (PLEG): container finished" podID="6cc9eaef-2205-4b44-926f-cb7799674915" containerID="9dba8f32f8cd956412436b2c75a1bdf66993a6a659305dcb5b6e0b5a992582e4" exitCode=0 Sep 11 12:20:42.757098 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:42.756666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" event={"ID":"6cc9eaef-2205-4b44-926f-cb7799674915","Type":"ContainerDied","Data":"9dba8f32f8cd956412436b2c75a1bdf66993a6a659305dcb5b6e0b5a992582e4"} Sep 11 12:20:43.763007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.762978 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" event={"ID":"5bac7e8a-3405-4eea-abc3-2b2f5b3685af","Type":"ContainerStarted","Data":"da0d63b146d5257176127fc2547738f10b09c4029e22a998c538d6ab044310c5"} Sep 11 12:20:43.887299 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.887277 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:43.945296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.945273 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util\") pod \"6cc9eaef-2205-4b44-926f-cb7799674915\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " Sep 11 12:20:43.945405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.945305 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle\") pod \"6cc9eaef-2205-4b44-926f-cb7799674915\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " Sep 11 12:20:43.945405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.945333 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppz9l\" (UniqueName: \"kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l\") pod \"6cc9eaef-2205-4b44-926f-cb7799674915\" (UID: \"6cc9eaef-2205-4b44-926f-cb7799674915\") " Sep 11 12:20:43.946000 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.945978 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle" (OuterVolumeSpecName: "bundle") pod "6cc9eaef-2205-4b44-926f-cb7799674915" (UID: "6cc9eaef-2205-4b44-926f-cb7799674915"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:43.947581 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.947562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l" (OuterVolumeSpecName: "kube-api-access-ppz9l") pod "6cc9eaef-2205-4b44-926f-cb7799674915" (UID: "6cc9eaef-2205-4b44-926f-cb7799674915"). InnerVolumeSpecName "kube-api-access-ppz9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:20:43.950211 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:43.950169 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util" (OuterVolumeSpecName: "util") pod "6cc9eaef-2205-4b44-926f-cb7799674915" (UID: "6cc9eaef-2205-4b44-926f-cb7799674915"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:20:44.046794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.046771 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:44.046794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.046793 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz9l\" (UniqueName: \"kubernetes.io/projected/6cc9eaef-2205-4b44-926f-cb7799674915-kube-api-access-ppz9l\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:44.046896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.046802 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6cc9eaef-2205-4b44-926f-cb7799674915-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:20:44.769450 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.769431 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" Sep 11 12:20:44.769796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.769451 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04t92q" event={"ID":"6cc9eaef-2205-4b44-926f-cb7799674915","Type":"ContainerDied","Data":"d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1"} Sep 11 12:20:44.769796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.769497 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75b3d08701861c67e4b588d270672e176f36bf8df48b285dc0b2651eac692f1" Sep 11 12:20:44.771310 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.771281 2575 generic.go:356] "Generic (PLEG): container finished" podID="5bac7e8a-3405-4eea-abc3-2b2f5b3685af" containerID="da0d63b146d5257176127fc2547738f10b09c4029e22a998c538d6ab044310c5" exitCode=0 Sep 11 12:20:44.771412 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:44.771346 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" event={"ID":"5bac7e8a-3405-4eea-abc3-2b2f5b3685af","Type":"ContainerDied","Data":"da0d63b146d5257176127fc2547738f10b09c4029e22a998c538d6ab044310c5"} Sep 11 12:20:45.779079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:45.779025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" event={"ID":"5bac7e8a-3405-4eea-abc3-2b2f5b3685af","Type":"ContainerStarted","Data":"f2c0c15f9bdc7a1c4f3609146dd0dd14cf668ef35c031bb6e3805bd7ad047852"} Sep 11 12:20:45.779497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:45.779103 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:20:45.804413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:45.804353 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" podStartSLOduration=2.208503062 podStartE2EDuration="6.804337362s" totalImagesPullingTime="3.542384564s" totalInitContainerRuntime="1.053449736s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:20:45.802868986 +0000 UTC m=+385.291074043" watchObservedRunningTime="2026-09-11 12:20:45.804337362 +0000 UTC m=+385.292542417" Sep 11 12:20:56.786204 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:20:56.786179 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8dd64745b-z59lw" Sep 11 12:21:00.034356 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034281 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g"] Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="util" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034589 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="util" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034597 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="pull" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034602 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="pull" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034632 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="extract" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034639 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="extract" Sep 11 12:21:00.034706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.034701 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc9eaef-2205-4b44-926f-cb7799674915" containerName="extract" Sep 11 12:21:00.042909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.042884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.045456 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.045437 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 12:21:00.045563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.045531 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sqz84\"" Sep 11 12:21:00.045639 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.045629 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 12:21:00.055463 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.055440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g"] Sep 11 12:21:00.161325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.161295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708f5282-ef7f-4758-a06e-a1b41a8366aa-tmp\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.161474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.161361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.161474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.161395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrbk\" (UniqueName: \"kubernetes.io/projected/708f5282-ef7f-4758-a06e-a1b41a8366aa-kube-api-access-zxrbk\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.262474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.262443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708f5282-ef7f-4758-a06e-a1b41a8366aa-tmp\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.262641 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.262497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.262641 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.262524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrbk\" (UniqueName: \"kubernetes.io/projected/708f5282-ef7f-4758-a06e-a1b41a8366aa-kube-api-access-zxrbk\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.262779 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:21:00.262655 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 11 12:21:00.262779 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:21:00.262739 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs podName:708f5282-ef7f-4758-a06e-a1b41a8366aa nodeName:}" failed. No retries permitted until 2026-09-11 12:21:00.762716892 +0000 UTC m=+400.250921937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs") pod "kube-auth-proxy-5b5f595c8f-lfx7g" (UID: "708f5282-ef7f-4758-a06e-a1b41a8366aa") : secret "kube-auth-proxy-tls" not found Sep 11 12:21:00.264784 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.264765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/708f5282-ef7f-4758-a06e-a1b41a8366aa-tmp\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.280584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.280563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrbk\" (UniqueName: \"kubernetes.io/projected/708f5282-ef7f-4758-a06e-a1b41a8366aa-kube-api-access-zxrbk\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.766238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.766212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.768577 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.768554 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/708f5282-ef7f-4758-a06e-a1b41a8366aa-tls-certs\") pod \"kube-auth-proxy-5b5f595c8f-lfx7g\" (UID: \"708f5282-ef7f-4758-a06e-a1b41a8366aa\") " pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:00.951922 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:00.951897 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" Sep 11 12:21:01.230435 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:01.230406 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g"] Sep 11 12:21:01.233814 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:21:01.233791 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f5282_ef7f_4758_a06e_a1b41a8366aa.slice/crio-4df464b760eb500a51bf970242943744b743b21024148838ea63c5d7f9e5eabc WatchSource:0}: Error finding container 4df464b760eb500a51bf970242943744b743b21024148838ea63c5d7f9e5eabc: Status 404 returned error can't find the container with id 4df464b760eb500a51bf970242943744b743b21024148838ea63c5d7f9e5eabc Sep 11 12:21:01.851718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:01.851679 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" event={"ID":"708f5282-ef7f-4758-a06e-a1b41a8366aa","Type":"ContainerStarted","Data":"4df464b760eb500a51bf970242943744b743b21024148838ea63c5d7f9e5eabc"} Sep 11 12:21:04.868476 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:04.868435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" event={"ID":"708f5282-ef7f-4758-a06e-a1b41a8366aa","Type":"ContainerStarted","Data":"302d7f53af833168e14fda8cd59cd9ac22bb2608c7962508681c526047dca5c6"} Sep 11 12:21:04.889061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:04.889014 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b5f595c8f-lfx7g" podStartSLOduration=3.0980637030000002 podStartE2EDuration="5.888999337s" totalImagesPullingTime="2.790935634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:04.887406888 +0000 UTC m=+404.375611955" watchObservedRunningTime="2026-09-11 12:21:04.888999337 +0000 UTC m=+404.377204392" Sep 11 12:21:06.677846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.677816 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7"] Sep 11 12:21:06.681338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.681315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.686023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.686002 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 12:21:06.686172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.686046 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5dmnl\"" Sep 11 12:21:06.712285 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.712257 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7"] Sep 11 12:21:06.813875 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.813990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.813990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ff791b-4607-4634-90e8-5f9d0da4dad6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.813990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rwn\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-kube-api-access-d2rwn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.813990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.813990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.813986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.814173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.814021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.814173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.814053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.814173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.814066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.914771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.914747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.914771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.914772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.914953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.914797 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.914953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.914931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.914993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ff791b-4607-4634-90e8-5f9d0da4dad6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rwn\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-kube-api-access-d2rwn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915490 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ff791b-4607-4634-90e8-5f9d0da4dad6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.915765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.915726 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.917173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.917152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.917304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.917288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.923482 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.923464 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.924308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.924292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rwn\" (UniqueName: \"kubernetes.io/projected/35ff791b-4607-4634-90e8-5f9d0da4dad6-kube-api-access-d2rwn\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7\" (UID: \"35ff791b-4607-4634-90e8-5f9d0da4dad6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:06.992380 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:06.992330 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:07.273341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:07.273297 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7"] Sep 11 12:21:07.275904 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:21:07.275879 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ff791b_4607_4634_90e8_5f9d0da4dad6.slice/crio-a19a34da2828b072e3f21d07b0463f3466cd77d5982b01d7968dd56102e08acf WatchSource:0}: Error finding container a19a34da2828b072e3f21d07b0463f3466cd77d5982b01d7968dd56102e08acf: Status 404 returned error can't find the container with id a19a34da2828b072e3f21d07b0463f3466cd77d5982b01d7968dd56102e08acf Sep 11 12:21:07.882467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:07.882437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" event={"ID":"35ff791b-4607-4634-90e8-5f9d0da4dad6","Type":"ContainerStarted","Data":"a19a34da2828b072e3f21d07b0463f3466cd77d5982b01d7968dd56102e08acf"} Sep 11 12:21:10.044334 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.044296 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:21:10.044726 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.044385 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:21:10.044726 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.044429 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:21:10.413097 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.413068 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-th5cm"] Sep 11 12:21:10.416787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.416772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.419731 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.419705 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 12:21:10.420144 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.419752 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-jzbbv\"" Sep 11 12:21:10.433153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.433131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-th5cm"] Sep 11 12:21:10.544981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.544954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-cert\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.545155 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.545003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bmz\" (UniqueName: \"kubernetes.io/projected/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-kube-api-access-f6bmz\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.649232 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.649201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bmz\" (UniqueName: \"kubernetes.io/projected/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-kube-api-access-f6bmz\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.649411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.649284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-cert\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.651908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.651886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-cert\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.657997 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.657966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bmz\" (UniqueName: \"kubernetes.io/projected/24a1f7dd-c7cf-42c2-a91d-7860c102b09b-kube-api-access-f6bmz\") pod \"odh-model-controller-7998c67dc4-th5cm\" (UID: \"24a1f7dd-c7cf-42c2-a91d-7860c102b09b\") " pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.726730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.726634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:10.825232 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.824773 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncct6"] Sep 11 12:21:10.829462 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.829441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:10.831992 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.831963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 12:21:10.832114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.832034 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9927d\"" Sep 11 12:21:10.838899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.838878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncct6"] Sep 11 12:21:10.901311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.901276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" event={"ID":"35ff791b-4607-4634-90e8-5f9d0da4dad6","Type":"ContainerStarted","Data":"b97f9fa4f5be30d3f70d95dfd8f8a899dc7f6a78cd36f802dd03f6d0db9d2aaa"} Sep 11 12:21:10.923916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.923869 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" podStartSLOduration=2.158127647 podStartE2EDuration="4.923852182s" totalImagesPullingTime="2.765724535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:10.922592711 +0000 UTC m=+410.410797767" watchObservedRunningTime="2026-09-11 12:21:10.923852182 +0000 UTC m=+410.412057301" Sep 11 12:21:10.954178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.952752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253e5021-baad-4b06-8d89-6937720fee17-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:10.954178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.952820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x4d\" (UniqueName: \"kubernetes.io/projected/253e5021-baad-4b06-8d89-6937720fee17-kube-api-access-k6x4d\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:10.993505 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:10.993439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:11.004413 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.004392 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-th5cm"] Sep 11 12:21:11.006936 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:21:11.006910 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a1f7dd_c7cf_42c2_a91d_7860c102b09b.slice/crio-a8dfeb07a101e5b47c646e4da2cbd44f9759b7fe5e8f2b1af28f9b87a4bd6c3e WatchSource:0}: Error finding container a8dfeb07a101e5b47c646e4da2cbd44f9759b7fe5e8f2b1af28f9b87a4bd6c3e: Status 404 returned error can't find the container with id a8dfeb07a101e5b47c646e4da2cbd44f9759b7fe5e8f2b1af28f9b87a4bd6c3e Sep 11 12:21:11.054033 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.054002 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x4d\" (UniqueName: \"kubernetes.io/projected/253e5021-baad-4b06-8d89-6937720fee17-kube-api-access-k6x4d\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:11.054375 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.054190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253e5021-baad-4b06-8d89-6937720fee17-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:11.056562 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.056543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/253e5021-baad-4b06-8d89-6937720fee17-tls-certs\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:11.067819 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.067794 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x4d\" (UniqueName: \"kubernetes.io/projected/253e5021-baad-4b06-8d89-6937720fee17-kube-api-access-k6x4d\") pod \"model-serving-api-54d89bd5fc-ncct6\" (UID: \"253e5021-baad-4b06-8d89-6937720fee17\") " pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:11.141825 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.141795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:11.416510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.416479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ncct6"] Sep 11 12:21:11.420348 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:21:11.420314 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253e5021_baad_4b06_8d89_6937720fee17.slice/crio-5279bde3dbb2097738307a9874dd5034db22c3f8db6260389589f2cfd638c0bb WatchSource:0}: Error finding container 5279bde3dbb2097738307a9874dd5034db22c3f8db6260389589f2cfd638c0bb: Status 404 returned error can't find the container with id 5279bde3dbb2097738307a9874dd5034db22c3f8db6260389589f2cfd638c0bb Sep 11 12:21:11.908098 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.908066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" event={"ID":"253e5021-baad-4b06-8d89-6937720fee17","Type":"ContainerStarted","Data":"5279bde3dbb2097738307a9874dd5034db22c3f8db6260389589f2cfd638c0bb"} Sep 11 12:21:11.909547 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.909521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" event={"ID":"24a1f7dd-c7cf-42c2-a91d-7860c102b09b","Type":"ContainerStarted","Data":"a8dfeb07a101e5b47c646e4da2cbd44f9759b7fe5e8f2b1af28f9b87a4bd6c3e"} Sep 11 12:21:11.999096 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:11.999064 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:12.917192 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:12.917120 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:12.918271 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:12.918051 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7" Sep 11 12:21:14.928721 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.928685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" event={"ID":"24a1f7dd-c7cf-42c2-a91d-7860c102b09b","Type":"ContainerStarted","Data":"f9d096d11ecfaf8a70b28e5cc3fc4e8768f0cc7f21381152278a48c1f09363f7"} Sep 11 12:21:14.929152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.928794 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:14.930659 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.930587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" event={"ID":"253e5021-baad-4b06-8d89-6937720fee17","Type":"ContainerStarted","Data":"8a2c883e8fe97dff180f8aa928830139137d06d45bc25fc8af0542fefc1aa806"} Sep 11 12:21:14.930842 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.930824 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:21:14.948385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.948341 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" podStartSLOduration=1.481715247 podStartE2EDuration="4.948327127s" totalImagesPullingTime="3.46661188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:14.947230408 +0000 UTC m=+414.435435460" watchObservedRunningTime="2026-09-11 12:21:14.948327127 +0000 UTC m=+414.436532183" Sep 11 12:21:14.963833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:14.963793 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" podStartSLOduration=1.959241118 podStartE2EDuration="4.963780056s" totalImagesPullingTime="3.004538938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:14.96253592 +0000 UTC m=+414.450740987" watchObservedRunningTime="2026-09-11 12:21:14.963780056 +0000 UTC m=+414.451985111" Sep 11 12:21:18.300804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.300775 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz"] Sep 11 12:21:18.311418 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.311389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.321355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.321334 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 12:21:18.321508 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.321404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 12:21:18.321777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.321758 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 12:21:18.322478 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.322462 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w7rrr\"" Sep 11 12:21:18.323456 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.323431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz"] Sep 11 12:21:18.420110 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ec635e-54ba-493d-97ff-c8bf598b4f57-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420202 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s74nt\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-kube-api-access-s74nt\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.420524 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.420510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ec635e-54ba-493d-97ff-c8bf598b4f57-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s74nt\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-kube-api-access-s74nt\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521853 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.521949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.521912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.522686 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.522657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.524074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.524043 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6ec635e-54ba-493d-97ff-c8bf598b4f57-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.524074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.524067 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.524241 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.524177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.524301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.524246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.534500 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.534475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s74nt\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-kube-api-access-s74nt\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.534605 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.534589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6ec635e-54ba-493d-97ff-c8bf598b4f57-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-5tbwz\" (UID: \"e6ec635e-54ba-493d-97ff-c8bf598b4f57\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.623655 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.623589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:18.907510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.907478 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz"] Sep 11 12:21:18.911515 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:21:18.911491 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ec635e_54ba_493d_97ff_c8bf598b4f57.slice/crio-d2146270c4478104f704551eb9e2c3bad75eda468b7e58babb13ae6fb57571e6 WatchSource:0}: Error finding container d2146270c4478104f704551eb9e2c3bad75eda468b7e58babb13ae6fb57571e6: Status 404 returned error can't find the container with id d2146270c4478104f704551eb9e2c3bad75eda468b7e58babb13ae6fb57571e6 Sep 11 12:21:18.952037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:18.952001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" event={"ID":"e6ec635e-54ba-493d-97ff-c8bf598b4f57","Type":"ContainerStarted","Data":"d2146270c4478104f704551eb9e2c3bad75eda468b7e58babb13ae6fb57571e6"} Sep 11 12:21:21.075039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:21.075003 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:21:21.075298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:21.075069 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:21:21.968274 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:21.968237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" event={"ID":"e6ec635e-54ba-493d-97ff-c8bf598b4f57","Type":"ContainerStarted","Data":"5b96019db8a18f86f6cef4d94600cc46de811eb0719bef9ac113153319937994"} Sep 11 12:21:21.968442 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:21.968380 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:22.000564 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:22.000515 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" podStartSLOduration=1.839260467 podStartE2EDuration="4.000501133s" totalImagesPullingTime="2.161240666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:21:21.997419297 +0000 UTC m=+421.485624352" watchObservedRunningTime="2026-09-11 12:21:22.000501133 +0000 UTC m=+421.488706189" Sep 11 12:21:22.977656 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:22.977603 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-5tbwz" Sep 11 12:21:25.937810 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:25.937785 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-th5cm" Sep 11 12:21:25.939822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:21:25.939801 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ncct6" Sep 11 12:22:20.898128 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:20.898102 2575 scope.go:117] "RemoveContainer" containerID="fb27028990654777a23a1c353ded1283d9f833939e87c494d3cb1a44b5a6053a" Sep 11 12:22:21.641541 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.641517 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp"] Sep 11 12:22:21.644762 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.644745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.647464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.647439 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 12:22:21.647587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.647462 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 12:22:21.648501 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.648485 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-h4m7h\"" Sep 11 12:22:21.653872 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.653854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp"] Sep 11 12:22:21.737987 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.737966 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l"] Sep 11 12:22:21.741180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.741167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.750876 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.750857 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l"] Sep 11 12:22:21.781448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.781428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.781535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.781472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppsdd\" (UniqueName: \"kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.781574 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.781547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.838313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.838293 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z"] Sep 11 12:22:21.841499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.841486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:21.849470 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.849453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z"] Sep 11 12:22:21.881871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.881847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.881969 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.881894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppsdd\" (UniqueName: \"kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.882044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlz2\" (UniqueName: \"kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.882082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.882125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.882162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.882355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.882389 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.882378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.892448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.892404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppsdd\" (UniqueName: \"kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.941349 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.941328 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z"] Sep 11 12:22:21.944755 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.944741 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:21.953650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.953634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:21.953864 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.953849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z"] Sep 11 12:22:21.983122 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:21.983228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983134 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:21.983228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8f7q\" (UniqueName: \"kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:21.983228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlz2\" (UniqueName: \"kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.983333 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.983376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.983620 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.983674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.983657 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:21.993455 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:21.993427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlz2\" (UniqueName: \"kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:22.049053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.049027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:22.084193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.084313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.084313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.084313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084263 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvd5m\" (UniqueName: \"kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.084313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8f7q\" (UniqueName: \"kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.084529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.084584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.084651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.084601 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.094132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.094051 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8f7q\" (UniqueName: \"kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.150023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.149968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:22.185258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.185226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvd5m\" (UniqueName: \"kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.185424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.185307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.185424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.185384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.186470 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.185781 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.186470 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.185904 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.195796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.195305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvd5m\" (UniqueName: \"kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.228444 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.228419 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp"] Sep 11 12:22:22.230592 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:22.230569 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c0341af_ff00_4a67_a6d5_a71312e592ed.slice/crio-40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66 WatchSource:0}: Error finding container 40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66: Status 404 returned error can't find the container with id 40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66 Sep 11 12:22:22.253976 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.253957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:22.255296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.255272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" event={"ID":"3c0341af-ff00-4a67-a6d5-a71312e592ed","Type":"ContainerStarted","Data":"40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66"} Sep 11 12:22:22.323854 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.323828 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l"] Sep 11 12:22:22.326406 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:22.326384 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030e122d_14cf_42a3_9148_5be75b0e61a0.slice/crio-a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707 WatchSource:0}: Error finding container a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707: Status 404 returned error can't find the container with id a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707 Sep 11 12:22:22.422510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.422485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z"] Sep 11 12:22:22.424031 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:22.424006 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26a073c_21f4_4ae8_aae6_b876a04375be.slice/crio-20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a WatchSource:0}: Error finding container 20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a: Status 404 returned error can't find the container with id 20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a Sep 11 12:22:22.540722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:22.540706 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z"] Sep 11 12:22:22.598305 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:22.598278 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550f4edc_714a_43f6_bdec_ba44e7b88edd.slice/crio-2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998 WatchSource:0}: Error finding container 2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998: Status 404 returned error can't find the container with id 2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998 Sep 11 12:22:23.262006 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.261980 2575 generic.go:356] "Generic (PLEG): container finished" podID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerID="cdf3d29750ee8e35b7bad09ecb40700973093ff094bd36ab5e55a6c5965442d8" exitCode=0 Sep 11 12:22:23.262323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.262035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" event={"ID":"3c0341af-ff00-4a67-a6d5-a71312e592ed","Type":"ContainerDied","Data":"cdf3d29750ee8e35b7bad09ecb40700973093ff094bd36ab5e55a6c5965442d8"} Sep 11 12:22:23.263734 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.263712 2575 generic.go:356] "Generic (PLEG): container finished" podID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerID="30731ad2ddf8739df20e38075e46ec99fb5045821e6185dad2945ac04d3c0468" exitCode=0 Sep 11 12:22:23.263854 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.263840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" event={"ID":"a26a073c-21f4-4ae8-aae6-b876a04375be","Type":"ContainerDied","Data":"30731ad2ddf8739df20e38075e46ec99fb5045821e6185dad2945ac04d3c0468"} Sep 11 12:22:23.263897 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.263872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" event={"ID":"a26a073c-21f4-4ae8-aae6-b876a04375be","Type":"ContainerStarted","Data":"20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a"} Sep 11 12:22:23.265744 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.265720 2575 generic.go:356] "Generic (PLEG): container finished" podID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerID="515fe552977d52973f2f9c0b05f36d711bc2eb915d722eeb16aecc97dc936d6b" exitCode=0 Sep 11 12:22:23.266020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.265940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" event={"ID":"030e122d-14cf-42a3-9148-5be75b0e61a0","Type":"ContainerDied","Data":"515fe552977d52973f2f9c0b05f36d711bc2eb915d722eeb16aecc97dc936d6b"} Sep 11 12:22:23.266020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.265990 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" event={"ID":"030e122d-14cf-42a3-9148-5be75b0e61a0","Type":"ContainerStarted","Data":"a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707"} Sep 11 12:22:23.267885 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.267864 2575 generic.go:356] "Generic (PLEG): container finished" podID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerID="a35045ed8c34642379bfc5c0887b54641642e00e212c329f3835b0ca7391619b" exitCode=0 Sep 11 12:22:23.267978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.267931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" event={"ID":"550f4edc-714a-43f6-bdec-ba44e7b88edd","Type":"ContainerDied","Data":"a35045ed8c34642379bfc5c0887b54641642e00e212c329f3835b0ca7391619b"} Sep 11 12:22:23.267978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:23.267963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" event={"ID":"550f4edc-714a-43f6-bdec-ba44e7b88edd","Type":"ContainerStarted","Data":"2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998"} Sep 11 12:22:24.280147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:24.280119 2575 generic.go:356] "Generic (PLEG): container finished" podID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerID="5a560fb19e18a9ebf9893a2295bfaa39adca9f9a76c053fd071161f2c642a756" exitCode=0 Sep 11 12:22:24.280442 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:24.280195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" event={"ID":"030e122d-14cf-42a3-9148-5be75b0e61a0","Type":"ContainerDied","Data":"5a560fb19e18a9ebf9893a2295bfaa39adca9f9a76c053fd071161f2c642a756"} Sep 11 12:22:25.286540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.286489 2575 generic.go:356] "Generic (PLEG): container finished" podID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerID="a0dcaef2e091b1f6474262d6e408d9473327f877d45ed2d005f293810ce9d3ae" exitCode=0 Sep 11 12:22:25.286879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.286558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" event={"ID":"3c0341af-ff00-4a67-a6d5-a71312e592ed","Type":"ContainerDied","Data":"a0dcaef2e091b1f6474262d6e408d9473327f877d45ed2d005f293810ce9d3ae"} Sep 11 12:22:25.288521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.288503 2575 generic.go:356] "Generic (PLEG): container finished" podID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerID="f2af7f03f0f7b78da4e2dc926d75ea48c6beb82c0568e5e29be9aea9e78d491d" exitCode=0 Sep 11 12:22:25.288604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.288580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" event={"ID":"a26a073c-21f4-4ae8-aae6-b876a04375be","Type":"ContainerDied","Data":"f2af7f03f0f7b78da4e2dc926d75ea48c6beb82c0568e5e29be9aea9e78d491d"} Sep 11 12:22:25.290883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.290861 2575 generic.go:356] "Generic (PLEG): container finished" podID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerID="a84ccf8a69ae34fe9f8be2b75ea66cd51603aa6ea67ed5faf52164b7e94de4bb" exitCode=0 Sep 11 12:22:25.290973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.290934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" event={"ID":"030e122d-14cf-42a3-9148-5be75b0e61a0","Type":"ContainerDied","Data":"a84ccf8a69ae34fe9f8be2b75ea66cd51603aa6ea67ed5faf52164b7e94de4bb"} Sep 11 12:22:25.292725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.292706 2575 generic.go:356] "Generic (PLEG): container finished" podID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerID="952cba029f700ed5e67e5a94dda388652ae2008ac67994c0fd8f1991fb7bd5a7" exitCode=0 Sep 11 12:22:25.292812 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:25.292768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" event={"ID":"550f4edc-714a-43f6-bdec-ba44e7b88edd","Type":"ContainerDied","Data":"952cba029f700ed5e67e5a94dda388652ae2008ac67994c0fd8f1991fb7bd5a7"} Sep 11 12:22:26.300695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.300636 2575 generic.go:356] "Generic (PLEG): container finished" podID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerID="6ce2eb28c33cd83e24b642c384c673ab8ae071d0bf434d80cc246943f799f03d" exitCode=0 Sep 11 12:22:26.301008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.300721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" event={"ID":"3c0341af-ff00-4a67-a6d5-a71312e592ed","Type":"ContainerDied","Data":"6ce2eb28c33cd83e24b642c384c673ab8ae071d0bf434d80cc246943f799f03d"} Sep 11 12:22:26.302834 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.302815 2575 generic.go:356] "Generic (PLEG): container finished" podID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerID="84163dc6d45e3a8a450234afbf633114e53bcacd4939cc5edff545ad2ffa5237" exitCode=0 Sep 11 12:22:26.302929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.302906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" event={"ID":"a26a073c-21f4-4ae8-aae6-b876a04375be","Type":"ContainerDied","Data":"84163dc6d45e3a8a450234afbf633114e53bcacd4939cc5edff545ad2ffa5237"} Sep 11 12:22:26.305056 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.305031 2575 generic.go:356] "Generic (PLEG): container finished" podID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerID="fc973355df132fa7af2a8427e8f5f049ac69a4845e0d3e3fdc6aa5e1d6710ac0" exitCode=0 Sep 11 12:22:26.305211 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.305100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" event={"ID":"550f4edc-714a-43f6-bdec-ba44e7b88edd","Type":"ContainerDied","Data":"fc973355df132fa7af2a8427e8f5f049ac69a4845e0d3e3fdc6aa5e1d6710ac0"} Sep 11 12:22:26.434196 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.434167 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:26.521091 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.521065 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util\") pod \"030e122d-14cf-42a3-9148-5be75b0e61a0\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " Sep 11 12:22:26.521224 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.521118 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vlz2\" (UniqueName: \"kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2\") pod \"030e122d-14cf-42a3-9148-5be75b0e61a0\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " Sep 11 12:22:26.521224 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.521142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle\") pod \"030e122d-14cf-42a3-9148-5be75b0e61a0\" (UID: \"030e122d-14cf-42a3-9148-5be75b0e61a0\") " Sep 11 12:22:26.521581 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.521559 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle" (OuterVolumeSpecName: "bundle") pod "030e122d-14cf-42a3-9148-5be75b0e61a0" (UID: "030e122d-14cf-42a3-9148-5be75b0e61a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:26.523137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.523112 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2" (OuterVolumeSpecName: "kube-api-access-6vlz2") pod "030e122d-14cf-42a3-9148-5be75b0e61a0" (UID: "030e122d-14cf-42a3-9148-5be75b0e61a0"). InnerVolumeSpecName "kube-api-access-6vlz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:22:26.526103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.526082 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util" (OuterVolumeSpecName: "util") pod "030e122d-14cf-42a3-9148-5be75b0e61a0" (UID: "030e122d-14cf-42a3-9148-5be75b0e61a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:26.622158 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.622103 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:26.622158 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.622130 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlz2\" (UniqueName: \"kubernetes.io/projected/030e122d-14cf-42a3-9148-5be75b0e61a0-kube-api-access-6vlz2\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:26.622158 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:26.622139 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/030e122d-14cf-42a3-9148-5be75b0e61a0-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.311661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.311600 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" Sep 11 12:22:27.311910 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.311655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rqg9l" event={"ID":"030e122d-14cf-42a3-9148-5be75b0e61a0","Type":"ContainerDied","Data":"a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707"} Sep 11 12:22:27.311910 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.311693 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b6b5e09c59fdebe1e204e65bbe465d416e657ebd75ea7d2bb9184f84cdd707" Sep 11 12:22:27.447514 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.446561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:27.484132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.484110 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:27.487100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.487080 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:27.528442 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.528416 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8f7q\" (UniqueName: \"kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q\") pod \"a26a073c-21f4-4ae8-aae6-b876a04375be\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " Sep 11 12:22:27.528566 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.528462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util\") pod \"a26a073c-21f4-4ae8-aae6-b876a04375be\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " Sep 11 12:22:27.528566 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.528510 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle\") pod \"a26a073c-21f4-4ae8-aae6-b876a04375be\" (UID: \"a26a073c-21f4-4ae8-aae6-b876a04375be\") " Sep 11 12:22:27.528988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.528964 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle" (OuterVolumeSpecName: "bundle") pod "a26a073c-21f4-4ae8-aae6-b876a04375be" (UID: "a26a073c-21f4-4ae8-aae6-b876a04375be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.530629 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.530584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q" (OuterVolumeSpecName: "kube-api-access-z8f7q") pod "a26a073c-21f4-4ae8-aae6-b876a04375be" (UID: "a26a073c-21f4-4ae8-aae6-b876a04375be"). InnerVolumeSpecName "kube-api-access-z8f7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:22:27.533626 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.533589 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util" (OuterVolumeSpecName: "util") pod "a26a073c-21f4-4ae8-aae6-b876a04375be" (UID: "a26a073c-21f4-4ae8-aae6-b876a04375be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.629832 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.629777 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle\") pod \"3c0341af-ff00-4a67-a6d5-a71312e592ed\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " Sep 11 12:22:27.629832 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.629807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle\") pod \"550f4edc-714a-43f6-bdec-ba44e7b88edd\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " Sep 11 12:22:27.629970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.629843 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util\") pod \"550f4edc-714a-43f6-bdec-ba44e7b88edd\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " Sep 11 12:22:27.629970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.629922 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvd5m\" (UniqueName: \"kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m\") pod \"550f4edc-714a-43f6-bdec-ba44e7b88edd\" (UID: \"550f4edc-714a-43f6-bdec-ba44e7b88edd\") " Sep 11 12:22:27.629970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.629960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppsdd\" (UniqueName: \"kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd\") pod \"3c0341af-ff00-4a67-a6d5-a71312e592ed\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " Sep 11 12:22:27.630118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630020 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util\") pod \"3c0341af-ff00-4a67-a6d5-a71312e592ed\" (UID: \"3c0341af-ff00-4a67-a6d5-a71312e592ed\") " Sep 11 12:22:27.630240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630224 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8f7q\" (UniqueName: \"kubernetes.io/projected/a26a073c-21f4-4ae8-aae6-b876a04375be-kube-api-access-z8f7q\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.630298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630247 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.630298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630260 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a26a073c-21f4-4ae8-aae6-b876a04375be-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.630492 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle" (OuterVolumeSpecName: "bundle") pod "3c0341af-ff00-4a67-a6d5-a71312e592ed" (UID: "3c0341af-ff00-4a67-a6d5-a71312e592ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.630587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.630574 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle" (OuterVolumeSpecName: "bundle") pod "550f4edc-714a-43f6-bdec-ba44e7b88edd" (UID: "550f4edc-714a-43f6-bdec-ba44e7b88edd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.632067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.632048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd" (OuterVolumeSpecName: "kube-api-access-ppsdd") pod "3c0341af-ff00-4a67-a6d5-a71312e592ed" (UID: "3c0341af-ff00-4a67-a6d5-a71312e592ed"). InnerVolumeSpecName "kube-api-access-ppsdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:22:27.632295 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.632275 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m" (OuterVolumeSpecName: "kube-api-access-pvd5m") pod "550f4edc-714a-43f6-bdec-ba44e7b88edd" (UID: "550f4edc-714a-43f6-bdec-ba44e7b88edd"). InnerVolumeSpecName "kube-api-access-pvd5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:22:27.635718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.635696 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util" (OuterVolumeSpecName: "util") pod "3c0341af-ff00-4a67-a6d5-a71312e592ed" (UID: "3c0341af-ff00-4a67-a6d5-a71312e592ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.636034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.636015 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util" (OuterVolumeSpecName: "util") pod "550f4edc-714a-43f6-bdec-ba44e7b88edd" (UID: "550f4edc-714a-43f6-bdec-ba44e7b88edd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:27.731518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731496 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvd5m\" (UniqueName: \"kubernetes.io/projected/550f4edc-714a-43f6-bdec-ba44e7b88edd-kube-api-access-pvd5m\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.731518 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731516 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppsdd\" (UniqueName: \"kubernetes.io/projected/3c0341af-ff00-4a67-a6d5-a71312e592ed-kube-api-access-ppsdd\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.731647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731524 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.731647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731532 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c0341af-ff00-4a67-a6d5-a71312e592ed-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.731647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731539 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-bundle\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:27.731647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:27.731545 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/550f4edc-714a-43f6-bdec-ba44e7b88edd-util\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:28.318457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.318432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" event={"ID":"3c0341af-ff00-4a67-a6d5-a71312e592ed","Type":"ContainerDied","Data":"40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66"} Sep 11 12:22:28.318457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.318446 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cwssbp" Sep 11 12:22:28.318887 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.318472 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f5172b42c050550eee9608d8324c42496398edfe2aa8e615b39e79e7117c66" Sep 11 12:22:28.320361 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.320346 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" Sep 11 12:22:28.320461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.320376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7trk8z" event={"ID":"a26a073c-21f4-4ae8-aae6-b876a04375be","Type":"ContainerDied","Data":"20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a"} Sep 11 12:22:28.320461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.320413 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20943dd6bec4c5de6534cf9ca55a4088cc37524230bd2431418b5266abb4867a" Sep 11 12:22:28.322330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.322308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" event={"ID":"550f4edc-714a-43f6-bdec-ba44e7b88edd","Type":"ContainerDied","Data":"2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998"} Sep 11 12:22:28.322421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.322335 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2447fce12a7979de7cffa31f41fb6651ce8e7dade32d7b9256a8c61c7e7fc998" Sep 11 12:22:28.322421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:28.322396 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5698z" Sep 11 12:22:35.432201 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432169 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb"] Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432487 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432496 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432504 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432515 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432520 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432531 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432537 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432547 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432552 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432559 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432564 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432570 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432575 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432581 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432586 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432592 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432596 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="util" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432602 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432606 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="pull" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432631 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432639 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="extract" Sep 11 12:22:35.432666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432644 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="extract" Sep 11 12:22:35.433320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432691 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c0341af-ff00-4a67-a6d5-a71312e592ed" containerName="extract" Sep 11 12:22:35.433320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432699 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="550f4edc-714a-43f6-bdec-ba44e7b88edd" containerName="extract" Sep 11 12:22:35.433320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432705 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="030e122d-14cf-42a3-9148-5be75b0e61a0" containerName="extract" Sep 11 12:22:35.433320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.432712 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a26a073c-21f4-4ae8-aae6-b876a04375be" containerName="extract" Sep 11 12:22:35.436871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.436856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.439465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.439445 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 12:22:35.439465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.439456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tt57v\"" Sep 11 12:22:35.439635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.439444 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 12:22:35.451160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.451131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb"] Sep 11 12:22:35.594427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.594401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqcp\" (UniqueName: \"kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.594529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.594446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.695399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.695346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqcp\" (UniqueName: \"kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.695399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.695392 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.695719 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.695704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.704029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.704007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqcp\" (UniqueName: \"kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:35.747135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:35.747113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:36.031529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:36.031506 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb"] Sep 11 12:22:36.033643 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:36.033598 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5122b456_eaa6_41e9_ae51_eab107938863.slice/crio-f0ae6ac50522637e93638bce8d2aa27ff58edf2915b5347828c51478333180a3 WatchSource:0}: Error finding container f0ae6ac50522637e93638bce8d2aa27ff58edf2915b5347828c51478333180a3: Status 404 returned error can't find the container with id f0ae6ac50522637e93638bce8d2aa27ff58edf2915b5347828c51478333180a3 Sep 11 12:22:36.365211 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:36.365115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" event={"ID":"5122b456-eaa6-41e9-ae51-eab107938863","Type":"ContainerStarted","Data":"f0ae6ac50522637e93638bce8d2aa27ff58edf2915b5347828c51478333180a3"} Sep 11 12:22:41.396907 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:41.396874 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" event={"ID":"5122b456-eaa6-41e9-ae51-eab107938863","Type":"ContainerStarted","Data":"2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86"} Sep 11 12:22:41.397314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:41.396957 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:41.427765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:41.427714 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" podStartSLOduration=1.91682215 podStartE2EDuration="6.427697078s" totalImagesPullingTime="4.510874928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:41.427481195 +0000 UTC m=+500.915686249" watchObservedRunningTime="2026-09-11 12:22:41.427697078 +0000 UTC m=+500.915902133" Sep 11 12:22:42.435272 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.435235 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn22m"] Sep 11 12:22:42.439229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.439207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:42.441745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.441727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8h6dp\"" Sep 11 12:22:42.456142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.456112 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn22m"] Sep 11 12:22:42.553508 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.553470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wrv\" (UniqueName: \"kubernetes.io/projected/19aa197f-9e2b-40d0-98b9-eaa0602df333-kube-api-access-99wrv\") pod \"authorino-operator-76b478bd78-sn22m\" (UID: \"19aa197f-9e2b-40d0-98b9-eaa0602df333\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:42.654833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.654808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wrv\" (UniqueName: \"kubernetes.io/projected/19aa197f-9e2b-40d0-98b9-eaa0602df333-kube-api-access-99wrv\") pod \"authorino-operator-76b478bd78-sn22m\" (UID: \"19aa197f-9e2b-40d0-98b9-eaa0602df333\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:42.663341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.663316 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wrv\" (UniqueName: \"kubernetes.io/projected/19aa197f-9e2b-40d0-98b9-eaa0602df333-kube-api-access-99wrv\") pod \"authorino-operator-76b478bd78-sn22m\" (UID: \"19aa197f-9e2b-40d0-98b9-eaa0602df333\") " pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:42.756308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:42.756252 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:43.028690 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:43.028667 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sn22m"] Sep 11 12:22:43.030477 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:43.030453 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19aa197f_9e2b_40d0_98b9_eaa0602df333.slice/crio-0b759f022c97c59b5c0ab9855f8b4c354247ae93afa3fa3788a65ea216240f02 WatchSource:0}: Error finding container 0b759f022c97c59b5c0ab9855f8b4c354247ae93afa3fa3788a65ea216240f02: Status 404 returned error can't find the container with id 0b759f022c97c59b5c0ab9855f8b4c354247ae93afa3fa3788a65ea216240f02 Sep 11 12:22:43.408574 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:43.408537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" event={"ID":"19aa197f-9e2b-40d0-98b9-eaa0602df333","Type":"ContainerStarted","Data":"0b759f022c97c59b5c0ab9855f8b4c354247ae93afa3fa3788a65ea216240f02"} Sep 11 12:22:45.419583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:45.419547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" event={"ID":"19aa197f-9e2b-40d0-98b9-eaa0602df333","Type":"ContainerStarted","Data":"d674723b1f1390429cf83815c418d573ff771e48133d1dd8b37bf5043e390e78"} Sep 11 12:22:45.419967 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:45.419642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:22:45.439129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:45.439084 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" podStartSLOduration=2.092219367 podStartE2EDuration="3.439070843s" totalImagesPullingTime="1.346851476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:45.436389835 +0000 UTC m=+504.924594888" watchObservedRunningTime="2026-09-11 12:22:45.439070843 +0000 UTC m=+504.927275897" Sep 11 12:22:52.404653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:52.404601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:54.580965 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.580935 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb"] Sep 11 12:22:54.581328 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.581106 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" podUID="5122b456-eaa6-41e9-ae51-eab107938863" containerName="manager" containerID="cri-o://2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86" gracePeriod=2 Sep 11 12:22:54.595194 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.595169 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb"] Sep 11 12:22:54.604804 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:22:54.604758 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5122b456-eaa6-41e9-ae51-eab107938863\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\": pods \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:54.612585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.612560 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:22:54.613106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.613062 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5122b456-eaa6-41e9-ae51-eab107938863" containerName="manager" Sep 11 12:22:54.613106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.613084 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5122b456-eaa6-41e9-ae51-eab107938863" containerName="manager" Sep 11 12:22:54.613266 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.613168 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5122b456-eaa6-41e9-ae51-eab107938863" containerName="manager" Sep 11 12:22:54.616370 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.616352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.639799 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.639774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:22:54.641953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.641924 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.642061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.641972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdfh\" (UniqueName: \"kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.658502 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:22:54.658469 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" podUID="5122b456-eaa6-41e9-ae51-eab107938863" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:54.743157 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.743116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.743157 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.743154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdfh\" (UniqueName: \"kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.743561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.743537 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.756765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.756738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdfh\" (UniqueName: \"kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-bv7xv\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:54.821668 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.821645 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:54.824411 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:22:54.824388 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-bqwlb\" is forbidden: User \"system:node:ip-10-0-140-239.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-239.ec2.internal' and this object" podUID="5122b456-eaa6-41e9-ae51-eab107938863" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:54.844037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.843995 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume\") pod \"5122b456-eaa6-41e9-ae51-eab107938863\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " Sep 11 12:22:54.844116 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.844039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlqcp\" (UniqueName: \"kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp\") pod \"5122b456-eaa6-41e9-ae51-eab107938863\" (UID: \"5122b456-eaa6-41e9-ae51-eab107938863\") " Sep 11 12:22:54.844448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.844424 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5122b456-eaa6-41e9-ae51-eab107938863" (UID: "5122b456-eaa6-41e9-ae51-eab107938863"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:22:54.846085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.846067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp" (OuterVolumeSpecName: "kube-api-access-vlqcp") pod "5122b456-eaa6-41e9-ae51-eab107938863" (UID: "5122b456-eaa6-41e9-ae51-eab107938863"). InnerVolumeSpecName "kube-api-access-vlqcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:22:54.945149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.945130 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5122b456-eaa6-41e9-ae51-eab107938863-extensions-socket-volume\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:54.945149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.945148 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlqcp\" (UniqueName: \"kubernetes.io/projected/5122b456-eaa6-41e9-ae51-eab107938863-kube-api-access-vlqcp\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:22:54.952579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.952559 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5122b456-eaa6-41e9-ae51-eab107938863" path="/var/lib/kubelet/pods/5122b456-eaa6-41e9-ae51-eab107938863/volumes" Sep 11 12:22:54.973729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:54.973708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:55.253203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.253177 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:22:55.253388 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:22:55.253366 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3790c438_5168_4d05_af98_99804565dec8.slice/crio-5050713554c4d2837ab2b6ed27aa8271112a599126dd0a874a4d603071994608 WatchSource:0}: Error finding container 5050713554c4d2837ab2b6ed27aa8271112a599126dd0a874a4d603071994608: Status 404 returned error can't find the container with id 5050713554c4d2837ab2b6ed27aa8271112a599126dd0a874a4d603071994608 Sep 11 12:22:55.473256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.473228 2575 generic.go:356] "Generic (PLEG): container finished" podID="5122b456-eaa6-41e9-ae51-eab107938863" containerID="2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86" exitCode=0 Sep 11 12:22:55.473376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.473292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bqwlb" Sep 11 12:22:55.473376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.473337 2575 scope.go:117] "RemoveContainer" containerID="2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86" Sep 11 12:22:55.475780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.475742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" event={"ID":"3790c438-5168-4d05-af98-99804565dec8","Type":"ContainerStarted","Data":"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc"} Sep 11 12:22:55.475880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.475800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" event={"ID":"3790c438-5168-4d05-af98-99804565dec8","Type":"ContainerStarted","Data":"5050713554c4d2837ab2b6ed27aa8271112a599126dd0a874a4d603071994608"} Sep 11 12:22:55.475972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.475955 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:22:55.487591 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.487571 2575 scope.go:117] "RemoveContainer" containerID="2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86" Sep 11 12:22:55.487918 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:22:55.487898 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86\": container with ID starting with 2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86 not found: ID does not exist" containerID="2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86" Sep 11 12:22:55.487981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.487938 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86"} err="failed to get container status \"2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86\": rpc error: code = NotFound desc = could not find container \"2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86\": container with ID starting with 2928d31e6f4656302941231640e31e5b6c0fc5ecdbd27a29cc0511c73280ce86 not found: ID does not exist" Sep 11 12:22:55.500945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:55.500874 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" podStartSLOduration=1.5008619570000001 podStartE2EDuration="1.500861957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:22:55.498302309 +0000 UTC m=+514.986507388" watchObservedRunningTime="2026-09-11 12:22:55.500861957 +0000 UTC m=+514.989067011" Sep 11 12:22:56.426684 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:22:56.426652 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-sn22m" Sep 11 12:23:06.484054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:06.484024 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:23:10.264784 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.264758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:23:10.265180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.264927 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" podUID="3790c438-5168-4d05-af98-99804565dec8" containerName="manager" containerID="cri-o://59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc" gracePeriod=10 Sep 11 12:23:10.507126 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.507103 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:23:10.550643 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.550557 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:23:10.551014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.550997 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3790c438-5168-4d05-af98-99804565dec8" containerName="manager" Sep 11 12:23:10.551014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.551013 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3790c438-5168-4d05-af98-99804565dec8" containerName="manager" Sep 11 12:23:10.551150 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.551083 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3790c438-5168-4d05-af98-99804565dec8" containerName="manager" Sep 11 12:23:10.553302 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.553279 2575 generic.go:356] "Generic (PLEG): container finished" podID="3790c438-5168-4d05-af98-99804565dec8" containerID="59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc" exitCode=0 Sep 11 12:23:10.553407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.553364 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" Sep 11 12:23:10.554404 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.554383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" event={"ID":"3790c438-5168-4d05-af98-99804565dec8","Type":"ContainerDied","Data":"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc"} Sep 11 12:23:10.554491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.554414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv" event={"ID":"3790c438-5168-4d05-af98-99804565dec8","Type":"ContainerDied","Data":"5050713554c4d2837ab2b6ed27aa8271112a599126dd0a874a4d603071994608"} Sep 11 12:23:10.554491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.554442 2575 scope.go:117] "RemoveContainer" containerID="59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc" Sep 11 12:23:10.554604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.554490 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.559625 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.559560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhdfh\" (UniqueName: \"kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh\") pod \"3790c438-5168-4d05-af98-99804565dec8\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " Sep 11 12:23:10.559723 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.559700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume\") pod \"3790c438-5168-4d05-af98-99804565dec8\" (UID: \"3790c438-5168-4d05-af98-99804565dec8\") " Sep 11 12:23:10.560156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.560136 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3790c438-5168-4d05-af98-99804565dec8" (UID: "3790c438-5168-4d05-af98-99804565dec8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:23:10.562324 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.562299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh" (OuterVolumeSpecName: "kube-api-access-nhdfh") pod "3790c438-5168-4d05-af98-99804565dec8" (UID: "3790c438-5168-4d05-af98-99804565dec8"). InnerVolumeSpecName "kube-api-access-nhdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:23:10.565846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.565820 2575 scope.go:117] "RemoveContainer" containerID="59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc" Sep 11 12:23:10.566236 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:23:10.566189 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc\": container with ID starting with 59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc not found: ID does not exist" containerID="59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc" Sep 11 12:23:10.566236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.566227 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc"} err="failed to get container status \"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc\": rpc error: code = NotFound desc = could not find container \"59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc\": container with ID starting with 59e04856cd43ea219987b85c821fb96caee97c6cae0762c29ab200f1c73d1ecc not found: ID does not exist" Sep 11 12:23:10.568055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.568033 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:23:10.660301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.660275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8gr\" (UniqueName: \"kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.660396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.660313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.660396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.660351 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3790c438-5168-4d05-af98-99804565dec8-extensions-socket-volume\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:23:10.660396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.660361 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdfh\" (UniqueName: \"kubernetes.io/projected/3790c438-5168-4d05-af98-99804565dec8-kube-api-access-nhdfh\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:23:10.761493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.761471 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8gr\" (UniqueName: \"kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.761579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.761509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.761914 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.761899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.771072 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.771052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8gr\" (UniqueName: \"kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dd889\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.872283 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.872262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:10.881815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.881793 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:23:10.887231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.887208 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-bv7xv"] Sep 11 12:23:10.957602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:10.957576 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3790c438-5168-4d05-af98-99804565dec8" path="/var/lib/kubelet/pods/3790c438-5168-4d05-af98-99804565dec8/volumes" Sep 11 12:23:11.148548 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:11.148528 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:23:11.151201 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:23:11.151179 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef24076a_b7e6_424d_ac7a_6569c254a1bd.slice/crio-af284534ad609896fa1b4b59ce2322aa52677116200d1e9de0bb337f83feeb86 WatchSource:0}: Error finding container af284534ad609896fa1b4b59ce2322aa52677116200d1e9de0bb337f83feeb86: Status 404 returned error can't find the container with id af284534ad609896fa1b4b59ce2322aa52677116200d1e9de0bb337f83feeb86 Sep 11 12:23:11.561593 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:11.561513 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerStarted","Data":"8a9cfd3b60adb8a8463c4b21778b13a99501ec0cd7382949242831f952f9ff4f"} Sep 11 12:23:11.561593 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:11.561570 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerStarted","Data":"af284534ad609896fa1b4b59ce2322aa52677116200d1e9de0bb337f83feeb86"} Sep 11 12:23:11.561981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:11.561693 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:11.584695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:11.584646 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" podStartSLOduration=1.5846295179999998 podStartE2EDuration="1.584629518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:11.583056071 +0000 UTC m=+531.071261127" watchObservedRunningTime="2026-09-11 12:23:11.584629518 +0000 UTC m=+531.072834567" Sep 11 12:23:22.569051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:22.569027 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:23:26.699772 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.699742 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq"] Sep 11 12:23:26.702803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.702780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.706146 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.706127 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b28bl\"" Sep 11 12:23:26.718143 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.718119 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq"] Sep 11 12:23:26.770821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.770795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.770936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.770841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8j5\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-kube-api-access-pm8j5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.770936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.770920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771050 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.770979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.771057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.771086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.771120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.771154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.771275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.771231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.871808 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.871779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.871935 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.871826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.871935 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.871844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.871959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8j5\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-kube-api-access-pm8j5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.871994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872418 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872462 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.872992 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.872967 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.874258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.874235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.874732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.874714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.881252 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.881232 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:26.881668 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:26.881651 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8j5\" (UniqueName: \"kubernetes.io/projected/2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2-kube-api-access-pm8j5\") pod \"maas-default-gateway-openshift-default-78bb78ff78-rq6dq\" (UID: \"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:27.012179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.012124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:27.287120 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.287093 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq"] Sep 11 12:23:27.288652 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:23:27.288627 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e16ce5e_204a_4a42_81b9_0adbcf4c7bd2.slice/crio-f36f8a44a3aba84316619f96374d33381cb3a05634bccc7c391c18abd6cffe09 WatchSource:0}: Error finding container f36f8a44a3aba84316619f96374d33381cb3a05634bccc7c391c18abd6cffe09: Status 404 returned error can't find the container with id f36f8a44a3aba84316619f96374d33381cb3a05634bccc7c391c18abd6cffe09 Sep 11 12:23:27.290780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.290748 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:23:27.290849 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.290810 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:23:27.290849 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.290837 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:23:27.645157 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.645119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" event={"ID":"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2","Type":"ContainerStarted","Data":"2505516a8df5c5dfe79ef27037b43235ae5050c5b1ab494946e235604c13f921"} Sep 11 12:23:27.645334 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.645180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" event={"ID":"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2","Type":"ContainerStarted","Data":"f36f8a44a3aba84316619f96374d33381cb3a05634bccc7c391c18abd6cffe09"} Sep 11 12:23:27.677935 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:27.677888 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" podStartSLOduration=1.6778732490000001 podStartE2EDuration="1.677873249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:23:27.674172543 +0000 UTC m=+547.162377598" watchObservedRunningTime="2026-09-11 12:23:27.677873249 +0000 UTC m=+547.166078303" Sep 11 12:23:28.012505 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:28.012442 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:28.016462 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:28.016440 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:28.650942 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:28.650913 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:23:28.651706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:23:28.651689 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:24:20.898026 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:24:20.898002 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:24:20.900786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:24:20.900767 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:25:01.153820 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.153781 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-ds9t5"] Sep 11 12:25:01.157162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.157145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.159578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.159554 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 12:25:01.159695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.159666 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j6ghs\"" Sep 11 12:25:01.169758 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.169657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-ds9t5"] Sep 11 12:25:01.276470 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.276438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.276697 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.276622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfl7b\" (UniqueName: \"kubernetes.io/projected/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-kube-api-access-kfl7b\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.377334 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.377306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfl7b\" (UniqueName: \"kubernetes.io/projected/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-kube-api-access-kfl7b\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.377478 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.377346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.377983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.377966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.386070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.386052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfl7b\" (UniqueName: \"kubernetes.io/projected/6f1b1984-c94f-46c7-bd68-2ad7398b1a19-kube-api-access-kfl7b\") pod \"payload-processing-5b54b7fbb7-ds9t5\" (UID: \"6f1b1984-c94f-46c7-bd68-2ad7398b1a19\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.467823 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.467765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:01.742714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.742692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-ds9t5"] Sep 11 12:25:01.744318 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:25:01.744292 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1b1984_c94f_46c7_bd68_2ad7398b1a19.slice/crio-063db58571bdeac9fe88abcffb5dc47f87568980bb8bb0b299bab4b5f80e60d5 WatchSource:0}: Error finding container 063db58571bdeac9fe88abcffb5dc47f87568980bb8bb0b299bab4b5f80e60d5: Status 404 returned error can't find the container with id 063db58571bdeac9fe88abcffb5dc47f87568980bb8bb0b299bab4b5f80e60d5 Sep 11 12:25:01.746301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:01.746282 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:25:02.115300 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:02.115216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" event={"ID":"6f1b1984-c94f-46c7-bd68-2ad7398b1a19","Type":"ContainerStarted","Data":"063db58571bdeac9fe88abcffb5dc47f87568980bb8bb0b299bab4b5f80e60d5"} Sep 11 12:25:04.129982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:04.129940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" event={"ID":"6f1b1984-c94f-46c7-bd68-2ad7398b1a19","Type":"ContainerStarted","Data":"87fa334a278c8ca2917494f7c1cbbb74cc42fcc63d1d7f4a3052b94fac52c065"} Sep 11 12:25:04.130368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:04.130024 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:04.150404 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:04.150356 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" podStartSLOduration=0.977106191 podStartE2EDuration="3.150343842s" totalImagesPullingTime="2.173237651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:04.147642296 +0000 UTC m=+643.635847347" watchObservedRunningTime="2026-09-11 12:25:04.150343842 +0000 UTC m=+643.638548897" Sep 11 12:25:10.163425 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.163392 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2" containerID="2505516a8df5c5dfe79ef27037b43235ae5050c5b1ab494946e235604c13f921" exitCode=0 Sep 11 12:25:10.163749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.163438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" event={"ID":"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2","Type":"ContainerDied","Data":"2505516a8df5c5dfe79ef27037b43235ae5050c5b1ab494946e235604c13f921"} Sep 11 12:25:10.163822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.163804 2575 scope.go:117] "RemoveContainer" containerID="2505516a8df5c5dfe79ef27037b43235ae5050c5b1ab494946e235604c13f921" Sep 11 12:25:10.164749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.164722 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:25:10.164846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.164829 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:25:10.164907 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:10.164881 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:25:11.170157 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:11.170124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" event={"ID":"2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2","Type":"ContainerStarted","Data":"0fbb109a11e420ed0253a5cb3c70b657ac8b528e940657909c9b5c6ab13eb1ae"} Sep 11 12:25:12.012497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:12.012468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:25:12.013521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:12.013504 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rq6dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 11 12:25:12.013586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:12.013547 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" podUID="2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 11 12:25:13.012803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:13.012772 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rq6dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 11 12:25:13.013243 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:13.012830 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" podUID="2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 11 12:25:13.651672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:13.651639 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:25:14.012756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:14.012668 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rq6dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 11 12:25:14.012756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:14.012730 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" podUID="2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 11 12:25:15.013078 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:15.013044 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-rq6dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 11 12:25:15.013513 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:15.013106 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" podUID="2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 11 12:25:15.137368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:15.137338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-ds9t5" Sep 11 12:25:16.016923 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:16.016897 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:25:16.018055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:16.018039 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-rq6dq" Sep 11 12:25:33.688875 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.688844 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v"] Sep 11 12:25:33.692748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.692726 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.696037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.696009 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 12:25:33.696135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.696083 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 12:25:33.696135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.696084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r6tj8\"" Sep 11 12:25:33.696223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.696084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 12:25:33.702707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.702687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v"] Sep 11 12:25:33.748295 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.748395 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.748395 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.748474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.748474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbb5\" (UniqueName: \"kubernetes.io/projected/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-kube-api-access-5sbb5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.748474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.748456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849136 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbb5\" (UniqueName: \"kubernetes.io/projected/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-kube-api-access-5sbb5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849237 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849237 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849321 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849321 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849321 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.849787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.849732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.851538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.851522 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.851667 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.851651 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:33.860081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:33.860059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbb5\" (UniqueName: \"kubernetes.io/projected/a8761947-ecb4-4dcf-8ae5-de8e5a65c114-kube-api-access-5sbb5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v\" (UID: \"a8761947-ecb4-4dcf-8ae5-de8e5a65c114\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:34.004717 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:34.004638 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:34.281446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:34.281423 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v"] Sep 11 12:25:34.282653 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:25:34.282597 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8761947_ecb4_4dcf_8ae5_de8e5a65c114.slice/crio-372172c8ecb66b1ddda3a2f3e39f52bce7b1d7e185695dfdb5aa344f53c1c9c8 WatchSource:0}: Error finding container 372172c8ecb66b1ddda3a2f3e39f52bce7b1d7e185695dfdb5aa344f53c1c9c8: Status 404 returned error can't find the container with id 372172c8ecb66b1ddda3a2f3e39f52bce7b1d7e185695dfdb5aa344f53c1c9c8 Sep 11 12:25:34.304520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:34.304496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" event={"ID":"a8761947-ecb4-4dcf-8ae5-de8e5a65c114","Type":"ContainerStarted","Data":"372172c8ecb66b1ddda3a2f3e39f52bce7b1d7e185695dfdb5aa344f53c1c9c8"} Sep 11 12:25:35.312902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:35.312864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" event={"ID":"a8761947-ecb4-4dcf-8ae5-de8e5a65c114","Type":"ContainerStarted","Data":"dbfa268e840db9238b5f6e2d93e693254310bc9c88a13ba3f4d5cdc0d21bb37a"} Sep 11 12:25:35.334023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:35.333978 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" podStartSLOduration=1.387514556 podStartE2EDuration="2.333961014s" totalImagesPullingTime="946.446458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:35.333024974 +0000 UTC m=+674.821230030" watchObservedRunningTime="2026-09-11 12:25:35.333961014 +0000 UTC m=+674.822166070" Sep 11 12:25:44.005302 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:44.005273 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:44.005715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:44.005320 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:44.016404 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:44.016382 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:44.370910 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:44.370847 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v" Sep 11 12:25:58.208760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.208732 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5"] Sep 11 12:25:58.213510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.213491 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.216406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.216387 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 11 12:25:58.228185 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.228161 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5"] Sep 11 12:25:58.330092 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjjdp\" (UniqueName: \"kubernetes.io/projected/d7f8fe7b-1c1b-474e-adbc-442511787ff4-kube-api-access-pjjdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.330092 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.330277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.330277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.330277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.330277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.330261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.430661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.430635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.430790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.430665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.430790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.430694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.430880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.430808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.430880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.430866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.431020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.431000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjjdp\" (UniqueName: \"kubernetes.io/projected/d7f8fe7b-1c1b-474e-adbc-442511787ff4-kube-api-access-pjjdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.431113 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.431038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.431170 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.431104 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.431225 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.431209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.433076 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.433057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7f8fe7b-1c1b-474e-adbc-442511787ff4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.433363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.433345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7f8fe7b-1c1b-474e-adbc-442511787ff4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.442154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.442132 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjjdp\" (UniqueName: \"kubernetes.io/projected/d7f8fe7b-1c1b-474e-adbc-442511787ff4-kube-api-access-pjjdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5\" (UID: \"d7f8fe7b-1c1b-474e-adbc-442511787ff4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.523250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.523180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:25:58.795601 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:58.795562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5"] Sep 11 12:25:58.797893 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:25:58.797862 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f8fe7b_1c1b_474e_adbc_442511787ff4.slice/crio-1ed0838f8511fa9a797d602395c5121bd71d5bf18883c98be7c4e7379a97e9ca WatchSource:0}: Error finding container 1ed0838f8511fa9a797d602395c5121bd71d5bf18883c98be7c4e7379a97e9ca: Status 404 returned error can't find the container with id 1ed0838f8511fa9a797d602395c5121bd71d5bf18883c98be7c4e7379a97e9ca Sep 11 12:25:59.438908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:59.438868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" event={"ID":"d7f8fe7b-1c1b-474e-adbc-442511787ff4","Type":"ContainerStarted","Data":"c2056461ad4aa46fba6e67c6df6d2eef3481de8fad2eac3f22baa5671be24866"} Sep 11 12:25:59.439249 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:59.438938 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" event={"ID":"d7f8fe7b-1c1b-474e-adbc-442511787ff4","Type":"ContainerStarted","Data":"1ed0838f8511fa9a797d602395c5121bd71d5bf18883c98be7c4e7379a97e9ca"} Sep 11 12:25:59.457417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:25:59.457369 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" podStartSLOduration=1.258815216 podStartE2EDuration="1.457354572s" totalImagesPullingTime="198.539356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:25:59.455891563 +0000 UTC m=+698.944096617" watchObservedRunningTime="2026-09-11 12:25:59.457354572 +0000 UTC m=+698.945559626" Sep 11 12:26:01.000738 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.000710 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc"] Sep 11 12:26:01.004319 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.004299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.006865 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.006840 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 11 12:26:01.014920 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.014899 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc"] Sep 11 12:26:01.052717 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.052695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.052892 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.052871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rhn\" (UniqueName: \"kubernetes.io/projected/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-kube-api-access-t7rhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.052944 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.052933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.053021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.052954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.053021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.052997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.053084 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.053041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.153955 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.153841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.153955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rhn\" (UniqueName: \"kubernetes.io/projected/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-kube-api-access-t7rhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154298 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154470 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.154964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.154891 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.156885 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.156863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.156972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.156958 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.163026 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.162995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rhn\" (UniqueName: \"kubernetes.io/projected/dcdf6376-cb0a-4fd0-96eb-538c4d7103d4-kube-api-access-t7rhn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-xvdfc\" (UID: \"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.317800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.317734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:01.594172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:01.594147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc"] Sep 11 12:26:01.596026 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:26:01.596000 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdf6376_cb0a_4fd0_96eb_538c4d7103d4.slice/crio-bd873b58cb05c451161e83a98fcc372710cce954ba7606959de01fab180da0d3 WatchSource:0}: Error finding container bd873b58cb05c451161e83a98fcc372710cce954ba7606959de01fab180da0d3: Status 404 returned error can't find the container with id bd873b58cb05c451161e83a98fcc372710cce954ba7606959de01fab180da0d3 Sep 11 12:26:02.457128 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:02.457090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" event={"ID":"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4","Type":"ContainerStarted","Data":"9763a8aeac6e8026d1a0bf8878038a9782cbaaae90d1f5f6092ed568dc3a5908"} Sep 11 12:26:02.457508 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:02.457141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" event={"ID":"dcdf6376-cb0a-4fd0-96eb-538c4d7103d4","Type":"ContainerStarted","Data":"bd873b58cb05c451161e83a98fcc372710cce954ba7606959de01fab180da0d3"} Sep 11 12:26:02.494574 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:02.494528 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" podStartSLOduration=2.301310823 podStartE2EDuration="2.49451469s" totalImagesPullingTime="193.203867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:26:02.49298583 +0000 UTC m=+701.981190885" watchObservedRunningTime="2026-09-11 12:26:02.49451469 +0000 UTC m=+701.982719744" Sep 11 12:26:08.524299 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:08.524268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:26:08.524783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:08.524319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:26:08.536443 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:08.536414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:26:09.512039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:09.512012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5" Sep 11 12:26:11.318291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:11.318260 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:11.318679 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:11.318312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:11.329810 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:11.329784 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:26:11.522340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:26:11.522319 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-xvdfc" Sep 11 12:27:56.510849 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.510780 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:27:56.514400 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.514375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.517437 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.517415 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1a8ee3-openshift-default-dockercfg-gr6wt\"" Sep 11 12:27:56.527449 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.527427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:27:56.618411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618453 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsbp\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.618855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.618796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.667265 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.667237 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:27:56.670917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.670895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.673364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.673346 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-38a724-openshift-default-dockercfg-nlrjf\"" Sep 11 12:27:56.687432 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.687405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:27:56.719972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.719952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.719977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsbp\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720557 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.720898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.720833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.722565 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.722543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.722929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.722913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.728891 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.728868 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.729224 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.729200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsbp\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp\") pod \"e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.820899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.820824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.820899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.820863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.820899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.820887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.820939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.820977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.821014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6b5\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.821036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.821109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.821147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.821128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.825294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.825271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6b5\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.928632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.929183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.928967 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.929237 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.929206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.930580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.929406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.930580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.929638 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.930580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.930204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.933371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.933324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.933839 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.933780 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.941007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.940961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.942407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.942210 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6b5\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5\") pod \"e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:56.981605 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:56.981579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:57.112957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.112927 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:27:57.114460 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:27:57.114433 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f1e127a_bab2_405d_bb5c_a0abc677c5b0.slice/crio-7a97572603cc2d873e325420b34d77c886752da6de04d83892fbb9ab6fa0c599 WatchSource:0}: Error finding container 7a97572603cc2d873e325420b34d77c886752da6de04d83892fbb9ab6fa0c599: Status 404 returned error can't find the container with id 7a97572603cc2d873e325420b34d77c886752da6de04d83892fbb9ab6fa0c599 Sep 11 12:27:57.116428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.116396 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:57.116532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.116470 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:57.116532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.116510 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:57.264054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.264021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:27:57.264665 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:27:57.264639 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8da46f_8e69_46d8_ae49_ab8a77d51024.slice/crio-b7cbf642e72047a3c593811f5bac91b1a3b50f7414ab5220c8fdd8f51358f45c WatchSource:0}: Error finding container b7cbf642e72047a3c593811f5bac91b1a3b50f7414ab5220c8fdd8f51358f45c: Status 404 returned error can't find the container with id b7cbf642e72047a3c593811f5bac91b1a3b50f7414ab5220c8fdd8f51358f45c Sep 11 12:27:57.267259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.267227 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:57.267343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.267300 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:57.267343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:57.267330 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:27:58.078916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.078883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerStarted","Data":"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7"} Sep 11 12:27:58.079363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.078976 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerStarted","Data":"7a97572603cc2d873e325420b34d77c886752da6de04d83892fbb9ab6fa0c599"} Sep 11 12:27:58.080853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.080828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerStarted","Data":"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99"} Sep 11 12:27:58.080968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.080874 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerStarted","Data":"b7cbf642e72047a3c593811f5bac91b1a3b50f7414ab5220c8fdd8f51358f45c"} Sep 11 12:27:58.105483 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.105432 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podStartSLOduration=2.105416035 podStartE2EDuration="2.105416035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:58.104171751 +0000 UTC m=+817.592376806" watchObservedRunningTime="2026-09-11 12:27:58.105416035 +0000 UTC m=+817.593621090" Sep 11 12:27:58.127434 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.127386 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podStartSLOduration=2.127367325 podStartE2EDuration="2.127367325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:27:58.126972976 +0000 UTC m=+817.615178045" watchObservedRunningTime="2026-09-11 12:27:58.127367325 +0000 UTC m=+817.615572391" Sep 11 12:27:58.825764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.825732 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:58.830654 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.830628 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:58.982643 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.982599 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:58.987094 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:58.987072 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:59.087426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:59.087364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:59.087818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:59.087750 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:27:59.088014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:59.087999 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:27:59.088102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:27:59.088042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:28:04.718485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.718453 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:28:04.723783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.723761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.727684 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.727662 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9226c5-openshift-default-dockercfg-s4nmg\"" Sep 11 12:28:04.739726 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.739692 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:28:04.906893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.906856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.906893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.906897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.906921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907118 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:04.907292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:04.907194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86cl\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86cl\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008860 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008860 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.008964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.008887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.009055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.009038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.009342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.009323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.010961 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.010933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.011222 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.011205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.015847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.015826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.016358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.016337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86cl\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl\") pod \"e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.038233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.038210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:05.332648 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.332597 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:28:05.334073 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:28:05.334052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801b682c_a4d3_44c8_9451_8921973d67ce.slice/crio-d1a566e45be688eac9e81db0e04e3e44f6e355e257dbf71c79fb7a2235abbcf6 WatchSource:0}: Error finding container d1a566e45be688eac9e81db0e04e3e44f6e355e257dbf71c79fb7a2235abbcf6: Status 404 returned error can't find the container with id d1a566e45be688eac9e81db0e04e3e44f6e355e257dbf71c79fb7a2235abbcf6 Sep 11 12:28:05.336524 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.336494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:05.336596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.336557 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:05.336596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:05.336585 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:06.140236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.140196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerStarted","Data":"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b"} Sep 11 12:28:06.140662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.140251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerStarted","Data":"d1a566e45be688eac9e81db0e04e3e44f6e355e257dbf71c79fb7a2235abbcf6"} Sep 11 12:28:06.161208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.161164 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podStartSLOduration=2.161148577 podStartE2EDuration="2.161148577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:06.159752607 +0000 UTC m=+825.647957655" watchObservedRunningTime="2026-09-11 12:28:06.161148577 +0000 UTC m=+825.649353632" Sep 11 12:28:06.243132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.243093 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:06.247340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.247317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.249998 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.249977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7e8d45b5-openshift-default-dockercfg-hwv72\"" Sep 11 12:28:06.260559 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.260538 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:06.321154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321264 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321306 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthll\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321407 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.321520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.321454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422531 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422531 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422753 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422753 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422925 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthll\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.422975 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.422995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.423033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.423252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.423682 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.423815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.423792 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.424994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.424974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.425386 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.425366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.434264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.434244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.434513 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.434495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthll\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll\") pod \"e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.558887 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.558866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:06.864583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.864560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:06.865047 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:28:06.865019 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56267f46_b935_488a_baff_9f9c796cec9b.slice/crio-b3f1269e88c02911aa365724855fc5bb0d59fbed21512cde0211763cfaca97ab WatchSource:0}: Error finding container b3f1269e88c02911aa365724855fc5bb0d59fbed21512cde0211763cfaca97ab: Status 404 returned error can't find the container with id b3f1269e88c02911aa365724855fc5bb0d59fbed21512cde0211763cfaca97ab Sep 11 12:28:06.867277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.867249 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:06.867343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.867312 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:06.867379 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:06.867353 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:07.039259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.039224 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:07.044467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.044445 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:07.147661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.147603 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerStarted","Data":"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2"} Sep 11 12:28:07.148093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.147691 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:07.148093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.147718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerStarted","Data":"b3f1269e88c02911aa365724855fc5bb0d59fbed21512cde0211763cfaca97ab"} Sep 11 12:28:07.149034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.149016 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:07.174109 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.174067 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podStartSLOduration=1.174053626 podStartE2EDuration="1.174053626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:07.170893354 +0000 UTC m=+826.659098421" watchObservedRunningTime="2026-09-11 12:28:07.174053626 +0000 UTC m=+826.662258679" Sep 11 12:28:07.560072 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.559989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:07.565734 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:07.565708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:08.158291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:08.158264 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:08.164067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:08.164039 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:15.128759 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:15.128718 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:28:15.129129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:15.128718 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:28:15.129129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:15.128786 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:28:15.129129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:15.128825 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:28:17.235561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.235522 2575 generic.go:356] "Generic (PLEG): container finished" podID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerID="017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99" exitCode=0 Sep 11 12:28:17.236083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.235680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerDied","Data":"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99"} Sep 11 12:28:17.236206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.236187 2575 scope.go:117] "RemoveContainer" containerID="017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99" Sep 11 12:28:17.237517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.237481 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:17.237653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.237572 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:17.237653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.237647 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:17.238098 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.238078 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerID="12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7" exitCode=0 Sep 11 12:28:17.238176 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.238134 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerDied","Data":"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7"} Sep 11 12:28:17.238467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.238450 2575 scope.go:117] "RemoveContainer" containerID="12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7" Sep 11 12:28:17.239475 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.239450 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:17.239558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.239520 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:17.239558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:17.239554 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:18.250870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.249810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerStarted","Data":"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5"} Sep 11 12:28:18.253754 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.253717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerStarted","Data":"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31"} Sep 11 12:28:18.828632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.825861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:28:18.828632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.825986 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:18.828632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.826031 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:18.982187 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.982150 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:28:18.982602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.982566 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 12:28:18.982723 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:18.982656 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 12:28:19.825923 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:19.825877 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:19.826438 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:19.825948 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:19.982508 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:19.982461 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 12:28:19.982736 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:19.982529 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 12:28:20.826869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:20.826384 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:20.826869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:20.826450 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:20.982071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:20.982040 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 12:28:20.982320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:20.982294 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 12:28:21.829637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:21.829043 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:21.829637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:21.829208 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:21.982258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:21.982209 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 12:28:21.982509 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:21.982356 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 12:28:22.826635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:22.826068 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:22.826635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:22.826133 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:22.982933 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:22.982896 2575 patch_prober.go:28] interesting pod/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Sep 11 12:28:22.983494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:22.983453 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Sep 11 12:28:23.825724 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:23.825678 2575 patch_prober.go:28] interesting pod/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Sep 11 12:28:23.825926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:23.825742 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Sep 11 12:28:24.000021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:23.999982 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:28:24.001287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.001259 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:28:24.005988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.005091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:28:24.199840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.199792 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:28:24.200045 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.199882 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:28:24.830038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.830008 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:28:24.830446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.830421 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:28:24.831321 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:24.831298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:28:25.748860 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.748824 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:28:25.751964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.751942 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.755108 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.755075 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-edffad-openshift-default-dockercfg-hj6rn\"" Sep 11 12:28:25.775640 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.775586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:28:25.824733 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.824901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.824901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkzvz\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.824901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.824901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.825086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.825086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.824983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.825086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.825054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.825199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.825101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkzvz\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.926914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.927496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.928110 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.928337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.928729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.928572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.931093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.928743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.933940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.933887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.947836 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.939856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkzvz\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.947836 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.942482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:25.951644 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:25.951577 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo\") pod \"e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:26.063858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.063764 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:26.325046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.324967 2575 generic.go:356] "Generic (PLEG): container finished" podID="56267f46-b935-488a-baff-9f9c796cec9b" containerID="194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2" exitCode=0 Sep 11 12:28:26.325228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.325049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerDied","Data":"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2"} Sep 11 12:28:26.325713 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.325693 2575 scope.go:117] "RemoveContainer" containerID="194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2" Sep 11 12:28:26.326929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.326900 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:26.327012 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.326964 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:26.327057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.327011 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:26.358554 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.358526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:28:26.358842 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:28:26.358812 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a4e37b_bd42_4ad2_bc86_88e0fcbd7394.slice/crio-ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee WatchSource:0}: Error finding container ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee: Status 404 returned error can't find the container with id ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee Sep 11 12:28:26.361171 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.361129 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:26.361283 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.361207 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:26.361283 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:26.361247 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:27.332454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.332378 2575 generic.go:356] "Generic (PLEG): container finished" podID="801b682c-a4d3-44c8-9451-8921973d67ce" containerID="84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b" exitCode=0 Sep 11 12:28:27.332454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.332443 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerDied","Data":"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b"} Sep 11 12:28:27.333012 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.332992 2575 scope.go:117] "RemoveContainer" containerID="84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b" Sep 11 12:28:27.334240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.334213 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:27.334363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.334285 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:27.334363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.334319 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:27.334655 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.334637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerStarted","Data":"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776"} Sep 11 12:28:27.336196 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.336178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerStarted","Data":"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521"} Sep 11 12:28:27.336305 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.336216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerStarted","Data":"ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee"} Sep 11 12:28:27.428078 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.428021 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podStartSLOduration=2.427998064 podStartE2EDuration="2.427998064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:27.420440762 +0000 UTC m=+846.908645818" watchObservedRunningTime="2026-09-11 12:28:27.427998064 +0000 UTC m=+846.916203124" Sep 11 12:28:27.559412 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.559313 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:27.559660 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.559637 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 12:28:27.559802 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:27.559781 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 12:28:28.063932 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.063904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:28.068868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.068843 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:28.343724 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.343624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerStarted","Data":"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f"} Sep 11 12:28:28.344230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.344212 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:28.344882 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.344861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:28.559426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.559390 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 12:28:28.559648 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:28.559455 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 12:28:29.039378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:29.039352 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:29.039581 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:29.039477 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:29.039581 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:29.039528 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:29.559553 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:29.559519 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 12:28:29.559938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:29.559571 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 12:28:30.038813 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:30.038778 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:30.039007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:30.038835 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:30.560228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:30.560195 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 12:28:30.560601 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:30.560249 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 12:28:31.039254 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:31.039222 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:31.039420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:31.039277 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:31.559934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:31.559890 2575 patch_prober.go:28] interesting pod/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 12:28:31.560103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:31.559950 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 12:28:32.039018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:32.038976 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:32.039480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:32.039032 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:32.562759 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:32.562730 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:32.563228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:32.563208 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:32.563818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:32.563803 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:33.039378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:33.039347 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:33.039764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:33.039409 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:34.039309 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:34.039274 2575 patch_prober.go:28] interesting pod/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 12:28:34.039496 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:34.039341 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 12:28:35.042403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:35.042371 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:35.042868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:35.042655 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:35.043308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:35.043292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:28:39.164658 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.164623 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt"] Sep 11 12:28:39.167549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.167534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:39.169982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.169960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:28:39.170103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.169960 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:28:39.181511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.181489 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt"] Sep 11 12:28:39.267537 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.267515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt\" (UID: \"3ceb6353-cd77-4373-a860-ec32080ac9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:39.368737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.368702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt\" (UID: \"3ceb6353-cd77-4373-a860-ec32080ac9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:39.369358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.369331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt\" (UID: \"3ceb6353-cd77-4373-a860-ec32080ac9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:39.477846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.477753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:39.769553 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:39.769525 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt"] Sep 11 12:28:39.771525 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:28:39.771495 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb6353_cd77_4373_a860_ec32080ac9dd.slice/crio-a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469 WatchSource:0}: Error finding container a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469: Status 404 returned error can't find the container with id a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469 Sep 11 12:28:40.418809 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:40.418777 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" event={"ID":"3ceb6353-cd77-4373-a860-ec32080ac9dd","Type":"ContainerStarted","Data":"a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469"} Sep 11 12:28:43.438891 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:43.438851 2575 generic.go:356] "Generic (PLEG): container finished" podID="3ceb6353-cd77-4373-a860-ec32080ac9dd" containerID="b1245d654e4fac13d80fd93af16fd17ab4c2433fe65eec8dfacc636e9b9f5590" exitCode=0 Sep 11 12:28:43.439244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:43.438899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" event={"ID":"3ceb6353-cd77-4373-a860-ec32080ac9dd","Type":"ContainerDied","Data":"b1245d654e4fac13d80fd93af16fd17ab4c2433fe65eec8dfacc636e9b9f5590"} Sep 11 12:28:44.587987 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:44.587957 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:44.737941 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:44.737869 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca\") pod \"3ceb6353-cd77-4373-a860-ec32080ac9dd\" (UID: \"3ceb6353-cd77-4373-a860-ec32080ac9dd\") " Sep 11 12:28:44.738187 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:44.738165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ceb6353-cd77-4373-a860-ec32080ac9dd" (UID: "3ceb6353-cd77-4373-a860-ec32080ac9dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:28:44.838868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:44.838838 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ceb6353-cd77-4373-a860-ec32080ac9dd-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:45.452334 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:45.452298 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" event={"ID":"3ceb6353-cd77-4373-a860-ec32080ac9dd","Type":"ContainerDied","Data":"a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469"} Sep 11 12:28:45.452512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:45.452354 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b6d385c44f77e29568a63857cc1c4f5b878d01ba2075b06521d65d33e7c469" Sep 11 12:28:45.452512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:45.452311 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt" Sep 11 12:28:48.473030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.472993 2575 generic.go:356] "Generic (PLEG): container finished" podID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerID="4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521" exitCode=0 Sep 11 12:28:48.473546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.473066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerDied","Data":"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521"} Sep 11 12:28:48.473546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.473497 2575 scope.go:117] "RemoveContainer" containerID="4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521" Sep 11 12:28:48.474675 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.474637 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:48.474801 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.474711 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:48.474801 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:48.474743 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:49.485712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:49.485678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerStarted","Data":"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600"} Sep 11 12:28:50.064840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.064810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:50.065008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.064946 2575 patch_prober.go:28] interesting pod/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 12:28:50.065008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.064989 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 12:28:50.458332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.458301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt"] Sep 11 12:28:50.460775 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.460753 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e8d45b5-vm7pt"] Sep 11 12:28:50.615457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.615419 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:50.616588 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.616552 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" containerID="cri-o://756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776" gracePeriod=30 Sep 11 12:28:50.953061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:50.953027 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ceb6353-cd77-4373-a860-ec32080ac9dd" path="/var/lib/kubelet/pods/3ceb6353-cd77-4373-a860-ec32080ac9dd/volumes" Sep 11 12:28:51.064426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.064395 2575 patch_prober.go:28] interesting pod/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 12:28:51.064651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.064460 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 12:28:51.590634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.586407 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:28:51.590634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.587144 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ceb6353-cd77-4373-a860-ec32080ac9dd" containerName="revoke-keys" Sep 11 12:28:51.590634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.587163 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ceb6353-cd77-4373-a860-ec32080ac9dd" containerName="revoke-keys" Sep 11 12:28:51.590634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.587298 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ceb6353-cd77-4373-a860-ec32080ac9dd" containerName="revoke-keys" Sep 11 12:28:51.616332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.611913 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.616332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.615169 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cba59841-gw-openshift-default-dockercfg-cq5jg\"" Sep 11 12:28:51.626320 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.626288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4vbz\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.721715 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.721631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822166 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4vbz\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822782 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822782 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822762 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.822890 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.822873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.823262 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.823234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.823342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.823245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.824603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.824581 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.824820 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.824803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.830502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.830480 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.830681 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.830663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4vbz\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz\") pod \"e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:51.931049 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:51.931015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:52.064959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.064932 2575 patch_prober.go:28] interesting pod/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 12:28:52.065080 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.064982 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 12:28:52.246824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.246798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:28:52.248773 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:28:52.248743 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d788efd_f93c_486d_a2cd_ab898bf8a8f1.slice/crio-6222b121a7dd1c7b3afbf3c20808de2e28ce0e192d80b74d1f035d1b0d73fd6d WatchSource:0}: Error finding container 6222b121a7dd1c7b3afbf3c20808de2e28ce0e192d80b74d1f035d1b0d73fd6d: Status 404 returned error can't find the container with id 6222b121a7dd1c7b3afbf3c20808de2e28ce0e192d80b74d1f035d1b0d73fd6d Sep 11 12:28:52.251282 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.251246 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:52.251367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.251337 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:52.251407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.251382 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:28:52.525323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.525243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerStarted","Data":"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462"} Sep 11 12:28:52.525323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.525292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerStarted","Data":"6222b121a7dd1c7b3afbf3c20808de2e28ce0e192d80b74d1f035d1b0d73fd6d"} Sep 11 12:28:52.549481 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.549427 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podStartSLOduration=1.549411047 podStartE2EDuration="1.549411047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:28:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:28:52.547673062 +0000 UTC m=+872.035878117" watchObservedRunningTime="2026-09-11 12:28:52.549411047 +0000 UTC m=+872.037616101" Sep 11 12:28:52.931141 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:52.931106 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:53.064544 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:53.064517 2575 patch_prober.go:28] interesting pod/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 12:28:53.064719 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:53.064567 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 12:28:53.936964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:53.936933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:54.064120 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:54.064093 2575 patch_prober.go:28] interesting pod/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Sep 11 12:28:54.064287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:54.064147 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Sep 11 12:28:54.537337 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:54.537308 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:54.538311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:54.538293 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:28:55.068260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.068234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:55.068603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.068488 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:55.069188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.069170 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:28:55.870563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.870542 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:55.962807 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962786 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.962948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962837 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.962948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.962948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962879 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rthll\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.962948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.963167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.962962 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.963167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963090 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.963167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963083 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data" (OuterVolumeSpecName: "istio-data") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:28:55.963167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963141 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.963167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert\") pod \"56267f46-b935-488a-baff-9f9c796cec9b\" (UID: \"56267f46-b935-488a-baff-9f9c796cec9b\") " Sep 11 12:28:55.963411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963287 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:28:55.963411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963385 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:28:55.963693 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:28:55.963850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963831 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:55.963931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963856 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:55.963931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963872 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:55.963931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.963888 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56267f46-b935-488a-baff-9f9c796cec9b-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:55.964093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.964037 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:28:55.965230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.965209 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token" (OuterVolumeSpecName: "istio-token") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:28:55.965331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.965310 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll" (OuterVolumeSpecName: "kube-api-access-rthll") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "kube-api-access-rthll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:28:55.965537 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.965519 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:28:55.965597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:55.965543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56267f46-b935-488a-baff-9f9c796cec9b" (UID: "56267f46-b935-488a-baff-9f9c796cec9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:28:56.064463 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.064437 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:56.064463 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.064459 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:56.064597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.064471 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthll\" (UniqueName: \"kubernetes.io/projected/56267f46-b935-488a-baff-9f9c796cec9b-kube-api-access-rthll\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:56.064597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.064484 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56267f46-b935-488a-baff-9f9c796cec9b-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:56.064597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.064495 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56267f46-b935-488a-baff-9f9c796cec9b-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:28:56.549945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.549920 2575 generic.go:356] "Generic (PLEG): container finished" podID="56267f46-b935-488a-baff-9f9c796cec9b" containerID="756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776" exitCode=0 Sep 11 12:28:56.550385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.549989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" Sep 11 12:28:56.550385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.550001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerDied","Data":"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776"} Sep 11 12:28:56.550385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.550061 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94" event={"ID":"56267f46-b935-488a-baff-9f9c796cec9b","Type":"ContainerDied","Data":"b3f1269e88c02911aa365724855fc5bb0d59fbed21512cde0211763cfaca97ab"} Sep 11 12:28:56.550385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.550092 2575 scope.go:117] "RemoveContainer" containerID="756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776" Sep 11 12:28:56.559950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.559933 2575 scope.go:117] "RemoveContainer" containerID="194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2" Sep 11 12:28:56.568235 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.568217 2575 scope.go:117] "RemoveContainer" containerID="756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776" Sep 11 12:28:56.568510 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:28:56.568491 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776\": container with ID starting with 756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776 not found: ID does not exist" containerID="756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776" Sep 11 12:28:56.568597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.568519 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776"} err="failed to get container status \"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776\": rpc error: code = NotFound desc = could not find container \"756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776\": container with ID starting with 756126eee74eb2bfa4588b6d57b3dd2eea085b19806ba9b9809eb707a6a4f776 not found: ID does not exist" Sep 11 12:28:56.568597 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.568535 2575 scope.go:117] "RemoveContainer" containerID="194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2" Sep 11 12:28:56.568794 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:28:56.568781 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2\": container with ID starting with 194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2 not found: ID does not exist" containerID="194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2" Sep 11 12:28:56.568832 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.568797 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2"} err="failed to get container status \"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2\": rpc error: code = NotFound desc = could not find container \"194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2\": container with ID starting with 194dde8dbfee4b64c10e6a2b9f58e2d624c18805c06d94ce914d394f355d27f2 not found: ID does not exist" Sep 11 12:28:56.575472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.575452 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:56.579311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.579290 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7e8d45b5-openshift-default-5cd68c64b4-82w94"] Sep 11 12:28:56.953292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:28:56.953264 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56267f46-b935-488a-baff-9f9c796cec9b" path="/var/lib/kubelet/pods/56267f46-b935-488a-baff-9f9c796cec9b/volumes" Sep 11 12:29:10.163901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.162540 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:44182->10.132.0.57:15021: read: connection reset by peer" start-of-body= Sep 11 12:29:10.163901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.162604 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:44182->10.132.0.57:15021: read: connection reset by peer" Sep 11 12:29:10.652846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.652820 2575 generic.go:356] "Generic (PLEG): container finished" podID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerID="b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462" exitCode=0 Sep 11 12:29:10.652982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.652878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerDied","Data":"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462"} Sep 11 12:29:10.653217 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.653204 2575 scope.go:117] "RemoveContainer" containerID="b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462" Sep 11 12:29:10.654370 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.654343 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:10.654431 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.654412 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:10.654469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:10.654443 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:11.660592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:11.660554 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerStarted","Data":"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7"} Sep 11 12:29:11.931746 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:11.931661 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:11.931909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:11.931847 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 12:29:11.931953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:11.931908 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 12:29:12.931742 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:12.931666 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 12:29:12.931742 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:12.931721 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 12:29:13.932303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:13.932274 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 12:29:13.932691 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:13.932321 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 12:29:14.932083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:14.932051 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 12:29:14.932252 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:14.932101 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 12:29:15.932162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:15.932127 2575 patch_prober.go:28] interesting pod/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 12:29:15.932601 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:15.932190 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 12:29:16.935401 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:16.935373 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:16.935803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:16.935783 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:16.936334 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:16.936315 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:18.939879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.939847 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz"] Sep 11 12:29:18.940270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940256 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.940270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940271 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.940347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940290 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.940347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940295 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.940406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940369 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.940406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.940379 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="56267f46-b935-488a-baff-9f9c796cec9b" containerName="istio-proxy" Sep 11 12:29:18.943371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.943355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:18.946820 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.946799 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:29:18.946947 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.946891 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:29:18.955059 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:18.955040 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz"] Sep 11 12:29:19.070246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.070215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz\" (UID: \"5a3c0c0f-b2e9-42d7-9629-47473955ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:19.171671 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.171640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz\" (UID: \"5a3c0c0f-b2e9-42d7-9629-47473955ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:19.172204 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.172186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz\" (UID: \"5a3c0c0f-b2e9-42d7-9629-47473955ef8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:19.254858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.254807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:19.526792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.526761 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz"] Sep 11 12:29:19.527329 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:29:19.527306 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3c0c0f_b2e9_42d7_9629_47473955ef8a.slice/crio-5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62 WatchSource:0}: Error finding container 5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62: Status 404 returned error can't find the container with id 5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62 Sep 11 12:29:19.715360 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:19.715333 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" event={"ID":"5a3c0c0f-b2e9-42d7-9629-47473955ef8a","Type":"ContainerStarted","Data":"5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62"} Sep 11 12:29:20.723390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:20.723363 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a3c0c0f-b2e9-42d7-9629-47473955ef8a" containerID="a2624fb9a55bdbd92294ca7d8924688f7cd7adf388064fb57a2d074c8399d0d7" exitCode=0 Sep 11 12:29:20.723772 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:20.723452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" event={"ID":"5a3c0c0f-b2e9-42d7-9629-47473955ef8a","Type":"ContainerDied","Data":"a2624fb9a55bdbd92294ca7d8924688f7cd7adf388064fb57a2d074c8399d0d7"} Sep 11 12:29:20.948814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:20.948787 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:29:20.949019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:20.948999 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:29:21.865115 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:21.865095 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:21.999381 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:21.999322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca\") pod \"5a3c0c0f-b2e9-42d7-9629-47473955ef8a\" (UID: \"5a3c0c0f-b2e9-42d7-9629-47473955ef8a\") " Sep 11 12:29:21.999645 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:21.999601 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a3c0c0f-b2e9-42d7-9629-47473955ef8a" (UID: "5a3c0c0f-b2e9-42d7-9629-47473955ef8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:29:22.100843 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:22.100820 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3c0c0f-b2e9-42d7-9629-47473955ef8a-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:22.737049 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:22.737022 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" event={"ID":"5a3c0c0f-b2e9-42d7-9629-47473955ef8a","Type":"ContainerDied","Data":"5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62"} Sep 11 12:29:22.737049 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:22.737048 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz" Sep 11 12:29:22.737227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:22.737073 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5274a96c8549302ed9b3daac48ff3e6d7a32a0631e74e095b7750ae3400a3e62" Sep 11 12:29:25.916944 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:25.916911 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz"] Sep 11 12:29:25.919457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:25.919436 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cba59841-k7gmz"] Sep 11 12:29:26.024172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.024138 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:29:26.024424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.024386 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" containerID="cri-o://7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7" gracePeriod=30 Sep 11 12:29:26.743081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.743054 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:29:26.743530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.743514 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3c0c0f-b2e9-42d7-9629-47473955ef8a" containerName="revoke-keys" Sep 11 12:29:26.743635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.743533 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3c0c0f-b2e9-42d7-9629-47473955ef8a" containerName="revoke-keys" Sep 11 12:29:26.743703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.743693 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3c0c0f-b2e9-42d7-9629-47473955ef8a" containerName="revoke-keys" Sep 11 12:29:26.747051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.747017 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.749896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.749863 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-74075c51-openshift-default-dockercfg-qw495\"" Sep 11 12:29:26.758764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.758740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:29:26.846089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846309 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846309 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846309 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846402 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846402 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.846464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.846430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdvz\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.951156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.951123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952551 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdvz\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952551 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952551 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952741 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.952741 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.952708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.953472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.953450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.953905 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.953887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.954239 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.954222 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.954720 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.954632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.955677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.955596 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.957645 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.957555 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3c0c0f-b2e9-42d7-9629-47473955ef8a" path="/var/lib/kubelet/pods/5a3c0c0f-b2e9-42d7-9629-47473955ef8a/volumes" Sep 11 12:29:26.957963 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.957943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.959240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.959226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.960216 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.960195 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:26.962099 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:26.962071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdvz\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz\") pod \"e2e-ait-74075c51-openshift-default-8459c7f856-f282s\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:27.057082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.057025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:27.335744 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.335721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:29:27.339368 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:29:27.339334 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c7b794_701d_4d29_a9c8_74521ec67e2d.slice/crio-f9537327aa6d3f7cc3b75f023b356b11798c69091a0940d6184b2b59394f815b WatchSource:0}: Error finding container f9537327aa6d3f7cc3b75f023b356b11798c69091a0940d6184b2b59394f815b: Status 404 returned error can't find the container with id f9537327aa6d3f7cc3b75f023b356b11798c69091a0940d6184b2b59394f815b Sep 11 12:29:27.343104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.343069 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:27.343208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.343145 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:27.343208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.343189 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:27.769602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.769572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerStarted","Data":"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9"} Sep 11 12:29:27.769789 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.769657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerStarted","Data":"f9537327aa6d3f7cc3b75f023b356b11798c69091a0940d6184b2b59394f815b"} Sep 11 12:29:27.795093 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:27.795052 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podStartSLOduration=1.7950395540000001 podStartE2EDuration="1.795039554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:27.793164051 +0000 UTC m=+907.281369106" watchObservedRunningTime="2026-09-11 12:29:27.795039554 +0000 UTC m=+907.283244657" Sep 11 12:29:28.057972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:28.057898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:28.062765 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:28.062740 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:28.775471 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:28.775433 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:28.776316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:28.776294 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:31.273993 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.273971 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:31.307338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307288 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307420 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307443 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307519 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4vbz\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307556 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307591 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307693 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:29:31.307832 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307774 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token\") pod \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\" (UID: \"1d788efd-f93c-486d-a2cd-ab898bf8a8f1\") " Sep 11 12:29:31.307904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307881 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:29:31.307966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:29:31.307966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307903 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:29:31.307966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.307952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:29:31.308421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.308345 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.308421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.308369 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.308421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.308382 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.308421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.308395 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.308421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.308408 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.309555 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.309528 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:29:31.309555 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.309539 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:29:31.309710 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.309649 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz" (OuterVolumeSpecName: "kube-api-access-k4vbz") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "kube-api-access-k4vbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:29:31.309957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.309941 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d788efd-f93c-486d-a2cd-ab898bf8a8f1" (UID: "1d788efd-f93c-486d-a2cd-ab898bf8a8f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:29:31.409672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.409645 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4vbz\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-kube-api-access-k4vbz\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.409672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.409669 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.409824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.409678 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.409824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.409686 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d788efd-f93c-486d-a2cd-ab898bf8a8f1-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:31.795534 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.795506 2575 generic.go:356] "Generic (PLEG): container finished" podID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerID="7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7" exitCode=0 Sep 11 12:29:31.795791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.795547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerDied","Data":"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7"} Sep 11 12:29:31.795791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.795555 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" Sep 11 12:29:31.795791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.795586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988" event={"ID":"1d788efd-f93c-486d-a2cd-ab898bf8a8f1","Type":"ContainerDied","Data":"6222b121a7dd1c7b3afbf3c20808de2e28ce0e192d80b74d1f035d1b0d73fd6d"} Sep 11 12:29:31.795791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.795605 2575 scope.go:117] "RemoveContainer" containerID="7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7" Sep 11 12:29:31.805649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.805631 2575 scope.go:117] "RemoveContainer" containerID="b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462" Sep 11 12:29:31.814444 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.814426 2575 scope.go:117] "RemoveContainer" containerID="7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7" Sep 11 12:29:31.814781 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:29:31.814754 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7\": container with ID starting with 7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7 not found: ID does not exist" containerID="7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7" Sep 11 12:29:31.814917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.814793 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7"} err="failed to get container status \"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7\": rpc error: code = NotFound desc = could not find container \"7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7\": container with ID starting with 7e2510d1cc5582b309b5b8623bc16b0984fda7957cab011c1cd2688606e7c9c7 not found: ID does not exist" Sep 11 12:29:31.814917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.814819 2575 scope.go:117] "RemoveContainer" containerID="b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462" Sep 11 12:29:31.815300 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:29:31.815277 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462\": container with ID starting with b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462 not found: ID does not exist" containerID="b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462" Sep 11 12:29:31.815440 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.815410 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462"} err="failed to get container status \"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462\": rpc error: code = NotFound desc = could not find container \"b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462\": container with ID starting with b53c5a9af182e73ae85b318ca357bb0bb8204de58686b6d37dccc328117dd462 not found: ID does not exist" Sep 11 12:29:31.818248 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.818231 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:29:31.823495 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:31.823475 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cba59841-gw-openshift-default-7bd46d6d8d-j6988"] Sep 11 12:29:32.952987 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:32.952956 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" path="/var/lib/kubelet/pods/1d788efd-f93c-486d-a2cd-ab898bf8a8f1/volumes" Sep 11 12:29:50.935274 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.935248 2575 generic.go:356] "Generic (PLEG): container finished" podID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerID="4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9" exitCode=0 Sep 11 12:29:50.935586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.935314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerDied","Data":"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9"} Sep 11 12:29:50.935748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.935724 2575 scope.go:117] "RemoveContainer" containerID="4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9" Sep 11 12:29:50.936843 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.936816 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:50.936953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.936900 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:50.937016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:50.936948 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:29:51.943087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:51.943049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerStarted","Data":"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af"} Sep 11 12:29:52.058051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:52.058021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:52.058223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:52.058194 2575 patch_prober.go:28] interesting pod/e2e-ait-74075c51-openshift-default-8459c7f856-f282s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 12:29:52.058274 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:52.058236 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 12:29:53.057308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:53.057278 2575 patch_prober.go:28] interesting pod/e2e-ait-74075c51-openshift-default-8459c7f856-f282s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 12:29:53.057676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:53.057334 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 12:29:54.057868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.057837 2575 patch_prober.go:28] interesting pod/e2e-ait-74075c51-openshift-default-8459c7f856-f282s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 12:29:54.058236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.057885 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 12:29:54.813480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.813445 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86"] Sep 11 12:29:54.813947 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.813929 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:29:54.814046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.813950 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:29:54.814046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.813988 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:29:54.814046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.813996 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:29:54.814188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.814101 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:29:54.817553 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.817532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:54.820106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.820086 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:29:54.820106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.820100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:29:54.825300 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.825275 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86"] Sep 11 12:29:54.932451 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:54.932419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74075c51-8xr86\" (UID: \"8521b4af-d7ac-42ae-915d-550770dad554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:55.033319 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.033293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74075c51-8xr86\" (UID: \"8521b4af-d7ac-42ae-915d-550770dad554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:55.033915 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.033898 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-74075c51-8xr86\" (UID: \"8521b4af-d7ac-42ae-915d-550770dad554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:55.057489 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.057463 2575 patch_prober.go:28] interesting pod/e2e-ait-74075c51-openshift-default-8459c7f856-f282s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 12:29:55.057571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.057504 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 12:29:55.130151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.130085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:55.443132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.443101 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86"] Sep 11 12:29:55.445007 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:29:55.444979 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8521b4af_d7ac_42ae_915d_550770dad554.slice/crio-8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8 WatchSource:0}: Error finding container 8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8: Status 404 returned error can't find the container with id 8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8 Sep 11 12:29:55.968022 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.967988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" event={"ID":"8521b4af-d7ac-42ae-915d-550770dad554","Type":"ContainerStarted","Data":"80b1eab98b5da495d6c7a1f6ccaec48d3d5ba5ee863f27ea7a5293fe376b8b58"} Sep 11 12:29:55.968149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.968047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" event={"ID":"8521b4af-d7ac-42ae-915d-550770dad554","Type":"ContainerStarted","Data":"8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8"} Sep 11 12:29:55.991190 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:55.991120 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" podStartSLOduration=1.991098301 podStartE2EDuration="1.991098301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:29:55.989790544 +0000 UTC m=+935.477995591" watchObservedRunningTime="2026-09-11 12:29:55.991098301 +0000 UTC m=+935.479303357" Sep 11 12:29:56.058057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:56.057995 2575 patch_prober.go:28] interesting pod/e2e-ait-74075c51-openshift-default-8459c7f856-f282s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 12:29:56.058057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:56.058048 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 12:29:56.979241 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:56.979200 2575 generic.go:356] "Generic (PLEG): container finished" podID="8521b4af-d7ac-42ae-915d-550770dad554" containerID="80b1eab98b5da495d6c7a1f6ccaec48d3d5ba5ee863f27ea7a5293fe376b8b58" exitCode=0 Sep 11 12:29:56.979790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:56.979296 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" event={"ID":"8521b4af-d7ac-42ae-915d-550770dad554","Type":"ContainerDied","Data":"80b1eab98b5da495d6c7a1f6ccaec48d3d5ba5ee863f27ea7a5293fe376b8b58"} Sep 11 12:29:57.063307 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:57.063282 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:57.063598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:57.063574 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:57.064382 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:57.064365 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:29:58.121990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.121970 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:58.267556 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.267497 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca\") pod \"8521b4af-d7ac-42ae-915d-550770dad554\" (UID: \"8521b4af-d7ac-42ae-915d-550770dad554\") " Sep 11 12:29:58.267818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.267799 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8521b4af-d7ac-42ae-915d-550770dad554" (UID: "8521b4af-d7ac-42ae-915d-550770dad554"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:29:58.368731 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.368710 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8521b4af-d7ac-42ae-915d-550770dad554-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:29:58.992010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.991987 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" Sep 11 12:29:58.992010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.991995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86" event={"ID":"8521b4af-d7ac-42ae-915d-550770dad554","Type":"ContainerDied","Data":"8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8"} Sep 11 12:29:58.992174 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:29:58.992036 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8340dcfea4bf97819f217844558eaa94f4eb9745dbc90f6e2355344a971a89c8" Sep 11 12:30:00.144405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.144374 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52"] Sep 11 12:30:00.144853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.144837 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8521b4af-d7ac-42ae-915d-550770dad554" containerName="revoke-keys" Sep 11 12:30:00.144896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.144855 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8521b4af-d7ac-42ae-915d-550770dad554" containerName="revoke-keys" Sep 11 12:30:00.144939 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.144930 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8521b4af-d7ac-42ae-915d-550770dad554" containerName="revoke-keys" Sep 11 12:30:00.144973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.144941 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d788efd-f93c-486d-a2cd-ab898bf8a8f1" containerName="istio-proxy" Sep 11 12:30:00.147949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.147930 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz"] Sep 11 12:30:00.148072 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.148046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" Sep 11 12:30:00.150409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.150385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:30:00.151585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.151572 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4"] Sep 11 12:30:00.151721 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.151694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" Sep 11 12:30:00.154866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.154846 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc"] Sep 11 12:30:00.154983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.154970 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" Sep 11 12:30:00.158761 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.158743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" Sep 11 12:30:00.161327 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.161309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz"] Sep 11 12:30:00.161899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.161883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" Sep 11 12:30:00.165274 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.165259 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52"] Sep 11 12:30:00.171186 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.169411 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc"] Sep 11 12:30:00.177661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.174352 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" Sep 11 12:30:00.177661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.176429 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7"] Sep 11 12:30:00.193942 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.189798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" Sep 11 12:30:00.193942 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.193430 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7"] Sep 11 12:30:00.195789 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.195772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" Sep 11 12:30:00.200835 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.200813 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4"] Sep 11 12:30:00.244470 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.244066 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" Sep 11 12:30:00.254649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.254579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" Sep 11 12:30:00.479569 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.479541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52"] Sep 11 12:30:00.480006 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:00.479983 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4aceeee_e25f_4234_b51e_57ef53cfb1a7.slice/crio-55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33 WatchSource:0}: Error finding container 55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33: Status 404 returned error can't find the container with id 55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33 Sep 11 12:30:00.495313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.495288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz"] Sep 11 12:30:00.496294 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:00.496274 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05628a32_a10c_4217_8b7b_9632cce21f7c.slice/crio-0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672 WatchSource:0}: Error finding container 0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672: Status 404 returned error can't find the container with id 0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672 Sep 11 12:30:00.535010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.534990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4"] Sep 11 12:30:00.536028 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:00.536002 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e45f27_a3f5_4033_83ee_e5cea42eba46.slice/crio-518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d WatchSource:0}: Error finding container 518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d: Status 404 returned error can't find the container with id 518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d Sep 11 12:30:00.561704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.561669 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7"] Sep 11 12:30:00.564314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.564293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc"] Sep 11 12:30:00.567772 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:00.567752 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb81d6_3eab_40af_9cdf_3d3f2e7814df.slice/crio-deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6 WatchSource:0}: Error finding container deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6: Status 404 returned error can't find the container with id deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6 Sep 11 12:30:00.568033 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:00.568014 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe9dbc0_a537_4976_ad88_ed12ee23e8d7.slice/crio-48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65 WatchSource:0}: Error finding container 48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65: Status 404 returned error can't find the container with id 48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65 Sep 11 12:30:00.994949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.994924 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86"] Sep 11 12:30:00.998433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:00.998409 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-74075c51-8xr86"] Sep 11 12:30:01.005247 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.005222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" event={"ID":"05628a32-a10c-4217-8b7b-9632cce21f7c","Type":"ContainerStarted","Data":"9fd1c3b7aed6f8444643a1414df6c5cd225081607157ca42e17c75be41da3da5"} Sep 11 12:30:01.005348 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.005270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" event={"ID":"05628a32-a10c-4217-8b7b-9632cce21f7c","Type":"ContainerStarted","Data":"0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672"} Sep 11 12:30:01.006854 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.006829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" event={"ID":"39e45f27-a3f5-4033-83ee-e5cea42eba46","Type":"ContainerStarted","Data":"c2c7f2109170ea790644e59c8987d3659cad952fd0ac927010e73025974b1990"} Sep 11 12:30:01.006960 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.006875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" event={"ID":"39e45f27-a3f5-4033-83ee-e5cea42eba46","Type":"ContainerStarted","Data":"518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d"} Sep 11 12:30:01.008449 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.008427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" event={"ID":"d4aceeee-e25f-4234-b51e-57ef53cfb1a7","Type":"ContainerStarted","Data":"9e5f48b5bf2af4e2ca53ab4c1139c6248746a6b0817aa008435696ca73a817f8"} Sep 11 12:30:01.008547 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.008464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" event={"ID":"d4aceeee-e25f-4234-b51e-57ef53cfb1a7","Type":"ContainerStarted","Data":"55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33"} Sep 11 12:30:01.010026 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.010008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" event={"ID":"d9cb81d6-3eab-40af-9cdf-3d3f2e7814df","Type":"ContainerStarted","Data":"9628798d957c8c47fe3ba196695c3626d34a1c8e45a55fe1a7d416431b06dabb"} Sep 11 12:30:01.010112 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.010044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" event={"ID":"d9cb81d6-3eab-40af-9cdf-3d3f2e7814df","Type":"ContainerStarted","Data":"deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6"} Sep 11 12:30:01.011454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.011437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" event={"ID":"2fe9dbc0-a537-4976-ad88-ed12ee23e8d7","Type":"ContainerStarted","Data":"71a7a1886ebfa39839eafe34ca895cadf9329d3bf1bb966052a135f51235539a"} Sep 11 12:30:01.011530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.011468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" event={"ID":"2fe9dbc0-a537-4976-ad88-ed12ee23e8d7","Type":"ContainerStarted","Data":"48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65"} Sep 11 12:30:01.024647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.024584 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" podStartSLOduration=1.024572465 podStartE2EDuration="1.024572465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:01.021794861 +0000 UTC m=+940.509999916" watchObservedRunningTime="2026-09-11 12:30:01.024572465 +0000 UTC m=+940.512777519" Sep 11 12:30:01.039250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.039212 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" podStartSLOduration=1.03920252 podStartE2EDuration="1.03920252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:01.038555101 +0000 UTC m=+940.526760156" watchObservedRunningTime="2026-09-11 12:30:01.03920252 +0000 UTC m=+940.527407574" Sep 11 12:30:01.061001 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.060964 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" podStartSLOduration=1.060948977 podStartE2EDuration="1.060948977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:01.05855177 +0000 UTC m=+940.546756826" watchObservedRunningTime="2026-09-11 12:30:01.060948977 +0000 UTC m=+940.549154069" Sep 11 12:30:01.110634 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.106753 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" podStartSLOduration=1.106732686 podStartE2EDuration="1.106732686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:01.101569511 +0000 UTC m=+940.589774566" watchObservedRunningTime="2026-09-11 12:30:01.106732686 +0000 UTC m=+940.594937742" Sep 11 12:30:01.119048 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:01.118990 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" podStartSLOduration=1.118967109 podStartE2EDuration="1.118967109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:01.11810049 +0000 UTC m=+940.606305544" watchObservedRunningTime="2026-09-11 12:30:01.118967109 +0000 UTC m=+940.607172167" Sep 11 12:30:02.019273 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.019243 2575 generic.go:356] "Generic (PLEG): container finished" podID="39e45f27-a3f5-4033-83ee-e5cea42eba46" containerID="c2c7f2109170ea790644e59c8987d3659cad952fd0ac927010e73025974b1990" exitCode=0 Sep 11 12:30:02.019604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.019299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" event={"ID":"39e45f27-a3f5-4033-83ee-e5cea42eba46","Type":"ContainerDied","Data":"c2c7f2109170ea790644e59c8987d3659cad952fd0ac927010e73025974b1990"} Sep 11 12:30:02.020933 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.020912 2575 generic.go:356] "Generic (PLEG): container finished" podID="d4aceeee-e25f-4234-b51e-57ef53cfb1a7" containerID="9e5f48b5bf2af4e2ca53ab4c1139c6248746a6b0817aa008435696ca73a817f8" exitCode=0 Sep 11 12:30:02.021004 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.020986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" event={"ID":"d4aceeee-e25f-4234-b51e-57ef53cfb1a7","Type":"ContainerDied","Data":"9e5f48b5bf2af4e2ca53ab4c1139c6248746a6b0817aa008435696ca73a817f8"} Sep 11 12:30:02.022532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.022512 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9cb81d6-3eab-40af-9cdf-3d3f2e7814df" containerID="9628798d957c8c47fe3ba196695c3626d34a1c8e45a55fe1a7d416431b06dabb" exitCode=0 Sep 11 12:30:02.022629 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.022583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" event={"ID":"d9cb81d6-3eab-40af-9cdf-3d3f2e7814df","Type":"ContainerDied","Data":"9628798d957c8c47fe3ba196695c3626d34a1c8e45a55fe1a7d416431b06dabb"} Sep 11 12:30:02.024115 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.024095 2575 generic.go:356] "Generic (PLEG): container finished" podID="2fe9dbc0-a537-4976-ad88-ed12ee23e8d7" containerID="71a7a1886ebfa39839eafe34ca895cadf9329d3bf1bb966052a135f51235539a" exitCode=0 Sep 11 12:30:02.024200 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.024120 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" event={"ID":"2fe9dbc0-a537-4976-ad88-ed12ee23e8d7","Type":"ContainerDied","Data":"71a7a1886ebfa39839eafe34ca895cadf9329d3bf1bb966052a135f51235539a"} Sep 11 12:30:02.025705 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.025685 2575 generic.go:356] "Generic (PLEG): container finished" podID="05628a32-a10c-4217-8b7b-9632cce21f7c" containerID="9fd1c3b7aed6f8444643a1414df6c5cd225081607157ca42e17c75be41da3da5" exitCode=0 Sep 11 12:30:02.025787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.025732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" event={"ID":"05628a32-a10c-4217-8b7b-9632cce21f7c","Type":"ContainerDied","Data":"9fd1c3b7aed6f8444643a1414df6c5cd225081607157ca42e17c75be41da3da5"} Sep 11 12:30:02.103690 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.103589 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:30:02.103867 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.103841 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" containerID="cri-o://628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af" gracePeriod=30 Sep 11 12:30:02.776584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.776539 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:02.780396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.780376 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.783195 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.783172 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-411110ed-openshift-default-dockercfg-fjn52\"" Sep 11 12:30:02.794147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.794124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:02.920010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.919974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920010 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920174 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.920445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.920361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm5z\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:02.962714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:02.962679 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8521b4af-d7ac-42ae-915d-550770dad554" path="/var/lib/kubelet/pods/8521b4af-d7ac-42ae-915d-550770dad554/volumes" Sep 11 12:30:03.021117 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm5z\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021748 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.021945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.021763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.022125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.022023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.023729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.023711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.023975 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.023960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.028937 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.028888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm5z\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.029664 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.029644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token\") pod \"e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.092208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.092168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:03.220876 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.220835 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" Sep 11 12:30:03.285092 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.285069 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" Sep 11 12:30:03.287126 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.287101 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" Sep 11 12:30:03.293403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.293385 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" Sep 11 12:30:03.295655 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.295597 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" Sep 11 12:30:03.410974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.410951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:03.412039 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:03.412014 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4779c8a4_a78d_44a3_b6d0_64c1ebb93206.slice/crio-1ccc7c5388a32d775e32ed64ba6b46125f415ce46b986a8c881e2a2d88ba168e WatchSource:0}: Error finding container 1ccc7c5388a32d775e32ed64ba6b46125f415ce46b986a8c881e2a2d88ba168e: Status 404 returned error can't find the container with id 1ccc7c5388a32d775e32ed64ba6b46125f415ce46b986a8c881e2a2d88ba168e Sep 11 12:30:03.414301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.414273 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:03.414360 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.414331 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:03.414391 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:03.414359 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:04.040943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.040918 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" Sep 11 12:30:04.041337 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.040932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz" event={"ID":"05628a32-a10c-4217-8b7b-9632cce21f7c","Type":"ContainerDied","Data":"0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672"} Sep 11 12:30:04.041337 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.040987 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a3399a464cc0417475ad8ae7ce8ca7b2f302a12ddf70e44e0c805fe3660d672" Sep 11 12:30:04.042942 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.042916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerStarted","Data":"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c"} Sep 11 12:30:04.043068 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.042964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerStarted","Data":"1ccc7c5388a32d775e32ed64ba6b46125f415ce46b986a8c881e2a2d88ba168e"} Sep 11 12:30:04.044329 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.044309 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" Sep 11 12:30:04.044427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.044337 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818830-lcsb4" event={"ID":"39e45f27-a3f5-4033-83ee-e5cea42eba46","Type":"ContainerDied","Data":"518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d"} Sep 11 12:30:04.044427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.044377 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518a626abaede3007abd205b6f0a2a75ffe0323e0ab66dcbf00dd1fa8e6e9d9d" Sep 11 12:30:04.045797 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.045773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" event={"ID":"d4aceeee-e25f-4234-b51e-57ef53cfb1a7","Type":"ContainerDied","Data":"55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33"} Sep 11 12:30:04.045888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.045796 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52" Sep 11 12:30:04.045888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.045808 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55502c7548f062c22d33e3f46e98df655a832ffba407bdc0454d1e00620cdc33" Sep 11 12:30:04.047231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.047211 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" Sep 11 12:30:04.047318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.047229 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7" event={"ID":"d9cb81d6-3eab-40af-9cdf-3d3f2e7814df","Type":"ContainerDied","Data":"deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6"} Sep 11 12:30:04.047318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.047270 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deef94f779151a8e7ef3f7df9c179a162d4226d053c2814e05b17fc9bddb0ca6" Sep 11 12:30:04.048779 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.048760 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" Sep 11 12:30:04.048872 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.048761 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc" event={"ID":"2fe9dbc0-a537-4976-ad88-ed12ee23e8d7","Type":"ContainerDied","Data":"48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65"} Sep 11 12:30:04.048921 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.048876 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48a999ba9588ceb89fcc699a74d71ad3925f612e8ec3e0bfee360c33a67eba65" Sep 11 12:30:04.066188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.066152 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podStartSLOduration=2.066139883 podStartE2EDuration="2.066139883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:04.063014981 +0000 UTC m=+943.551220039" watchObservedRunningTime="2026-09-11 12:30:04.066139883 +0000 UTC m=+943.554344937" Sep 11 12:30:04.092811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.092787 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:04.097281 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:04.097262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:05.055330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:05.055305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:05.056296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:05.056279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:06.316697 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.316661 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r"] Sep 11 12:30:06.317138 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317125 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05628a32-a10c-4217-8b7b-9632cce21f7c" containerName="cleanup" Sep 11 12:30:06.317189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317141 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="05628a32-a10c-4217-8b7b-9632cce21f7c" containerName="cleanup" Sep 11 12:30:06.317189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317151 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4aceeee-e25f-4234-b51e-57ef53cfb1a7" containerName="cleanup" Sep 11 12:30:06.317189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317155 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4aceeee-e25f-4234-b51e-57ef53cfb1a7" containerName="cleanup" Sep 11 12:30:06.317189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317172 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e45f27-a3f5-4033-83ee-e5cea42eba46" containerName="cleanup" Sep 11 12:30:06.317189 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317180 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e45f27-a3f5-4033-83ee-e5cea42eba46" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317198 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fe9dbc0-a537-4976-ad88-ed12ee23e8d7" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317203 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fe9dbc0-a537-4976-ad88-ed12ee23e8d7" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317211 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9cb81d6-3eab-40af-9cdf-3d3f2e7814df" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317216 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9cb81d6-3eab-40af-9cdf-3d3f2e7814df" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317275 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="05628a32-a10c-4217-8b7b-9632cce21f7c" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317283 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4aceeee-e25f-4234-b51e-57ef53cfb1a7" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317289 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9cb81d6-3eab-40af-9cdf-3d3f2e7814df" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317296 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fe9dbc0-a537-4976-ad88-ed12ee23e8d7" containerName="cleanup" Sep 11 12:30:06.317331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.317301 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e45f27-a3f5-4033-83ee-e5cea42eba46" containerName="cleanup" Sep 11 12:30:06.320665 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.320646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:06.323162 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.323145 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:30:06.323255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.323164 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:30:06.328827 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.328798 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r"] Sep 11 12:30:06.464015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.463990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9226c5-mkt6r\" (UID: \"226521c4-f3b8-4e5a-8e32-d6c2939a96c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:06.564755 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.564719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9226c5-mkt6r\" (UID: \"226521c4-f3b8-4e5a-8e32-d6c2939a96c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:06.565354 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.565337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9226c5-mkt6r\" (UID: \"226521c4-f3b8-4e5a-8e32-d6c2939a96c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:06.633580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.633519 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:06.907190 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:06.907163 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r"] Sep 11 12:30:06.908544 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:30:06.908519 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226521c4_f3b8_4e5a_8e32_d6c2939a96c8.slice/crio-ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db WatchSource:0}: Error finding container ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db: Status 404 returned error can't find the container with id ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db Sep 11 12:30:07.069856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.069831 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" event={"ID":"226521c4-f3b8-4e5a-8e32-d6c2939a96c8","Type":"ContainerStarted","Data":"ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db"} Sep 11 12:30:07.343722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.343701 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:30:07.474865 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.474773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.474866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.474902 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.474934 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475032 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.474971 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntdvz\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475076 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475148 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert\") pod \"18c7b794-701d-4d29-a9c8-74521ec67e2d\" (UID: \"18c7b794-701d-4d29-a9c8-74521ec67e2d\") " Sep 11 12:30:07.475438 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475361 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:07.475520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475494 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data" (OuterVolumeSpecName: "istio-data") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:07.475656 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475636 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:07.475776 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475754 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.475935 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475914 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.476034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475941 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.476034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475766 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:07.476034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.475873 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:30:07.477232 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.477210 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:30:07.477698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.477671 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz" (OuterVolumeSpecName: "kube-api-access-ntdvz") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "kube-api-access-ntdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:30:07.478137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.478120 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:07.478278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.478250 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token" (OuterVolumeSpecName: "istio-token") pod "18c7b794-701d-4d29-a9c8-74521ec67e2d" (UID: "18c7b794-701d-4d29-a9c8-74521ec67e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:30:07.576911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576886 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c7b794-701d-4d29-a9c8-74521ec67e2d-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.576911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576911 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.577074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576919 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.577074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576927 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.577074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576934 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:07.577074 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:07.576941 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntdvz\" (UniqueName: \"kubernetes.io/projected/18c7b794-701d-4d29-a9c8-74521ec67e2d-kube-api-access-ntdvz\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:08.079472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.079433 2575 generic.go:356] "Generic (PLEG): container finished" podID="226521c4-f3b8-4e5a-8e32-d6c2939a96c8" containerID="062c4b8dcfc4aa7536c5e895b11fad964422a96d4a475c77473b975760a2ec4a" exitCode=0 Sep 11 12:30:08.079671 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.079517 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" event={"ID":"226521c4-f3b8-4e5a-8e32-d6c2939a96c8","Type":"ContainerDied","Data":"062c4b8dcfc4aa7536c5e895b11fad964422a96d4a475c77473b975760a2ec4a"} Sep 11 12:30:08.081263 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.081240 2575 generic.go:356] "Generic (PLEG): container finished" podID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerID="628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af" exitCode=0 Sep 11 12:30:08.081397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.081286 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" Sep 11 12:30:08.081397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.081334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerDied","Data":"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af"} Sep 11 12:30:08.081397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.081386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s" event={"ID":"18c7b794-701d-4d29-a9c8-74521ec67e2d","Type":"ContainerDied","Data":"f9537327aa6d3f7cc3b75f023b356b11798c69091a0940d6184b2b59394f815b"} Sep 11 12:30:08.081554 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.081417 2575 scope.go:117] "RemoveContainer" containerID="628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af" Sep 11 12:30:08.094379 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.094362 2575 scope.go:117] "RemoveContainer" containerID="4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9" Sep 11 12:30:08.104087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.104067 2575 scope.go:117] "RemoveContainer" containerID="628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af" Sep 11 12:30:08.104350 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:30:08.104330 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af\": container with ID starting with 628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af not found: ID does not exist" containerID="628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af" Sep 11 12:30:08.104433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.104362 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af"} err="failed to get container status \"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af\": rpc error: code = NotFound desc = could not find container \"628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af\": container with ID starting with 628200cf1179a1a83c45d45013ffce1490e8926ef1f78b19d84c15c01d8959af not found: ID does not exist" Sep 11 12:30:08.104433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.104382 2575 scope.go:117] "RemoveContainer" containerID="4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9" Sep 11 12:30:08.104647 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:30:08.104622 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9\": container with ID starting with 4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9 not found: ID does not exist" containerID="4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9" Sep 11 12:30:08.104692 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.104653 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9"} err="failed to get container status \"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9\": rpc error: code = NotFound desc = could not find container \"4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9\": container with ID starting with 4faa33993eb676db859428bbb55da566f1a6819bf15382e546ae0ca42a6c20c9 not found: ID does not exist" Sep 11 12:30:08.112129 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.112108 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:30:08.117953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.117933 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-74075c51-openshift-default-8459c7f856-f282s"] Sep 11 12:30:08.953797 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:08.953766 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" path="/var/lib/kubelet/pods/18c7b794-701d-4d29-a9c8-74521ec67e2d/volumes" Sep 11 12:30:09.238452 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:09.238421 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:09.398255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:09.398219 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca\") pod \"226521c4-f3b8-4e5a-8e32-d6c2939a96c8\" (UID: \"226521c4-f3b8-4e5a-8e32-d6c2939a96c8\") " Sep 11 12:30:09.398555 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:09.398534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "226521c4-f3b8-4e5a-8e32-d6c2939a96c8" (UID: "226521c4-f3b8-4e5a-8e32-d6c2939a96c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:30:09.500038 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:09.499955 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/226521c4-f3b8-4e5a-8e32-d6c2939a96c8-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:10.097303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:10.097272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" event={"ID":"226521c4-f3b8-4e5a-8e32-d6c2939a96c8","Type":"ContainerDied","Data":"ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db"} Sep 11 12:30:10.097303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:10.097294 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r" Sep 11 12:30:10.097712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:10.097314 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecbcdf5bdf87861be5ff086624732f4832a4cd254a33c4106dc378b22e9174db" Sep 11 12:30:17.246159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:17.246113 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc"] Sep 11 12:30:17.248984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:17.248953 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-9226c5-29818830-zxvmc"] Sep 11 12:30:17.465206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:17.465173 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r"] Sep 11 12:30:17.468114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:17.468086 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9226c5-mkt6r"] Sep 11 12:30:18.953152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:18.953122 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226521c4-f3b8-4e5a-8e32-d6c2939a96c8" path="/var/lib/kubelet/pods/226521c4-f3b8-4e5a-8e32-d6c2939a96c8/volumes" Sep 11 12:30:18.953506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:18.953379 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fe9dbc0-a537-4976-ad88-ed12ee23e8d7" path="/var/lib/kubelet/pods/2fe9dbc0-a537-4976-ad88-ed12ee23e8d7/volumes" Sep 11 12:30:29.226364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.226331 2575 generic.go:356] "Generic (PLEG): container finished" podID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerID="4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c" exitCode=0 Sep 11 12:30:29.226784 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.226405 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerDied","Data":"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c"} Sep 11 12:30:29.226844 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.226829 2575 scope.go:117] "RemoveContainer" containerID="4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c" Sep 11 12:30:29.227991 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.227968 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:29.228067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.228032 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:29.228067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:29.228062 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:30:30.236223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.236179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerStarted","Data":"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d"} Sep 11 12:30:30.273123 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273087 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6"] Sep 11 12:30:30.273778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273754 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226521c4-f3b8-4e5a-8e32-d6c2939a96c8" containerName="revoke-keys" Sep 11 12:30:30.273778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273780 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="226521c4-f3b8-4e5a-8e32-d6c2939a96c8" containerName="revoke-keys" Sep 11 12:30:30.273950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273805 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:30:30.273950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273814 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:30:30.273950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273930 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="226521c4-f3b8-4e5a-8e32-d6c2939a96c8" containerName="revoke-keys" Sep 11 12:30:30.273950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273943 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:30:30.274148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.273953 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:30:30.289419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.289383 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6"] Sep 11 12:30:30.289589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.289507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:30.293756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.293335 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:30:30.293756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.293588 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:30:30.411674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.411639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-411110ed-dm5s6\" (UID: \"4ba8003e-c11b-40f4-8363-0c364b68379e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:30.512300 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.512214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-411110ed-dm5s6\" (UID: \"4ba8003e-c11b-40f4-8363-0c364b68379e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:30.512881 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.512860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-411110ed-dm5s6\" (UID: \"4ba8003e-c11b-40f4-8363-0c364b68379e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:30.607164 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.607129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:30.885275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:30.883716 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6"] Sep 11 12:30:31.094127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.094079 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:31.094592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.094307 2575 patch_prober.go:28] interesting pod/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 12:30:31.094592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.094365 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 12:30:31.244132 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.244104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" event={"ID":"4ba8003e-c11b-40f4-8363-0c364b68379e","Type":"ContainerStarted","Data":"f01f99d04f08c5f9934cd2b3ea0b02f62ec3f026006f5b4abdf57966d2565d21"} Sep 11 12:30:31.244459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.244160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" event={"ID":"4ba8003e-c11b-40f4-8363-0c364b68379e","Type":"ContainerStarted","Data":"85f76783a680951d022fb92e069bb7f8b0f7bf5a61f7452133716541014646b3"} Sep 11 12:30:31.262545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:31.262505 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" podStartSLOduration=1.262493905 podStartE2EDuration="1.262493905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:30:31.261171839 +0000 UTC m=+970.749376905" watchObservedRunningTime="2026-09-11 12:30:31.262493905 +0000 UTC m=+970.750698959" Sep 11 12:30:32.093000 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:32.092900 2575 patch_prober.go:28] interesting pod/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 12:30:32.093000 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:32.092964 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 12:30:32.251503 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:32.251469 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ba8003e-c11b-40f4-8363-0c364b68379e" containerID="f01f99d04f08c5f9934cd2b3ea0b02f62ec3f026006f5b4abdf57966d2565d21" exitCode=0 Sep 11 12:30:32.251900 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:32.251522 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" event={"ID":"4ba8003e-c11b-40f4-8363-0c364b68379e","Type":"ContainerDied","Data":"f01f99d04f08c5f9934cd2b3ea0b02f62ec3f026006f5b4abdf57966d2565d21"} Sep 11 12:30:33.093218 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.093184 2575 patch_prober.go:28] interesting pod/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 12:30:33.093487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.093461 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 12:30:33.446494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.446471 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:33.540429 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.540397 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca\") pod \"4ba8003e-c11b-40f4-8363-0c364b68379e\" (UID: \"4ba8003e-c11b-40f4-8363-0c364b68379e\") " Sep 11 12:30:33.540806 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.540780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ba8003e-c11b-40f4-8363-0c364b68379e" (UID: "4ba8003e-c11b-40f4-8363-0c364b68379e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:30:33.641723 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:33.641695 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ba8003e-c11b-40f4-8363-0c364b68379e-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:34.093271 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:34.093220 2575 patch_prober.go:28] interesting pod/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 12:30:34.093471 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:34.093287 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 12:30:34.266102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:34.266071 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" Sep 11 12:30:34.266259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:34.266104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6" event={"ID":"4ba8003e-c11b-40f4-8363-0c364b68379e","Type":"ContainerDied","Data":"85f76783a680951d022fb92e069bb7f8b0f7bf5a61f7452133716541014646b3"} Sep 11 12:30:34.266259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:34.266174 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f76783a680951d022fb92e069bb7f8b0f7bf5a61f7452133716541014646b3" Sep 11 12:30:35.055543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:35.055513 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:35.097164 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:35.097136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:35.273563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:35.273535 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:37.498015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:37.497983 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6"] Sep 11 12:30:37.502815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:37.502789 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-411110ed-dm5s6"] Sep 11 12:30:37.660125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:37.660087 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:37.660372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:37.660336 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" containerID="cri-o://a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d" gracePeriod=30 Sep 11 12:30:38.952794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:38.952764 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba8003e-c11b-40f4-8363-0c364b68379e" path="/var/lib/kubelet/pods/4ba8003e-c11b-40f4-8363-0c364b68379e/volumes" Sep 11 12:30:42.913707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:42.913685 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:43.040870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.040792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.040870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.040852 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.040870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.040876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041024 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041050 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:43.041114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041065 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041129 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041155 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:43.041268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041192 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkm5z\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041221 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token\") pod \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\" (UID: \"4779c8a4-a78d-44a3-b6d0-64c1ebb93206\") " Sep 11 12:30:43.041535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041280 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:30:43.041535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:43.041535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041376 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data" (OuterVolumeSpecName: "istio-data") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:43.041732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041670 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.041732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041690 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.041732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041703 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.041732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041716 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.041732 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.041731 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.043431 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.043405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token" (OuterVolumeSpecName: "istio-token") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:30:43.043543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.043489 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z" (OuterVolumeSpecName: "kube-api-access-wkm5z") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "kube-api-access-wkm5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:30:43.043764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.043747 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:30:43.043844 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.043827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4779c8a4-a78d-44a3-b6d0-64c1ebb93206" (UID: "4779c8a4-a78d-44a3-b6d0-64c1ebb93206"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:30:43.142680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.142653 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.142680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.142676 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkm5z\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-kube-api-access-wkm5z\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.142833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.142687 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.142833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.142696 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4779c8a4-a78d-44a3-b6d0-64c1ebb93206-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:30:43.329704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.329597 2575 generic.go:356] "Generic (PLEG): container finished" podID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerID="a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d" exitCode=0 Sep 11 12:30:43.329704 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.329683 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerDied","Data":"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d"} Sep 11 12:30:43.329934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.329744 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" Sep 11 12:30:43.329934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.329751 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f" event={"ID":"4779c8a4-a78d-44a3-b6d0-64c1ebb93206","Type":"ContainerDied","Data":"1ccc7c5388a32d775e32ed64ba6b46125f415ce46b986a8c881e2a2d88ba168e"} Sep 11 12:30:43.329934 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.329782 2575 scope.go:117] "RemoveContainer" containerID="a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d" Sep 11 12:30:43.340483 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.340463 2575 scope.go:117] "RemoveContainer" containerID="4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c" Sep 11 12:30:43.349219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.349201 2575 scope.go:117] "RemoveContainer" containerID="a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d" Sep 11 12:30:43.349489 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:30:43.349472 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d\": container with ID starting with a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d not found: ID does not exist" containerID="a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d" Sep 11 12:30:43.349535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.349500 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d"} err="failed to get container status \"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d\": rpc error: code = NotFound desc = could not find container \"a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d\": container with ID starting with a83a6da817922815fcd38ab5795c2acba9d169fdf2fb35168daf8288c1acf55d not found: ID does not exist" Sep 11 12:30:43.349535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.349516 2575 scope.go:117] "RemoveContainer" containerID="4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c" Sep 11 12:30:43.349794 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:30:43.349777 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c\": container with ID starting with 4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c not found: ID does not exist" containerID="4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c" Sep 11 12:30:43.349839 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.349800 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c"} err="failed to get container status \"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c\": rpc error: code = NotFound desc = could not find container \"4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c\": container with ID starting with 4efd235b901d88ba6017dd98ac3de7ddd37d60c17bc6dd76fcb4a6cf35c4484c not found: ID does not exist" Sep 11 12:30:43.357731 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.357710 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:43.360388 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:43.360368 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-411110ed-openshift-default-775b7687d4-fzb5f"] Sep 11 12:30:44.953778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:30:44.953746 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" path="/var/lib/kubelet/pods/4779c8a4-a78d-44a3-b6d0-64c1ebb93206/volumes" Sep 11 12:31:07.816487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:07.816453 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:31:07.817419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:07.816709 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" containerID="cri-o://a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f" gracePeriod=30 Sep 11 12:31:08.714786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.714753 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:31:08.715454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715427 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba8003e-c11b-40f4-8363-0c364b68379e" containerName="revoke-keys" Sep 11 12:31:08.715454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715453 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba8003e-c11b-40f4-8363-0c364b68379e" containerName="revoke-keys" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715473 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715481 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="18c7b794-701d-4d29-a9c8-74521ec67e2d" containerName="istio-proxy" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715508 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715515 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715531 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:08.715669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715539 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:08.715994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715686 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba8003e-c11b-40f4-8363-0c364b68379e" containerName="revoke-keys" Sep 11 12:31:08.715994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.715706 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:08.719176 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.719156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.721922 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.721900 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-82ead5-openshift-default-dockercfg-rnms6\"" Sep 11 12:31:08.734484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.734462 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:31:08.772528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8fk4\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772848 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772903 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.772955 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.773012 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.772964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873771 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.873970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8fk4\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.874340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.873977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.874340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.874049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.874446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.874385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.875927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.874707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.875927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.874735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.876561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.876538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.879737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.879715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.882373 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.882345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8fk4\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:08.882766 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:08.882749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token\") pod \"e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:09.029815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.029737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:09.307602 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.307577 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:31:09.308540 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:31:09.308512 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec284f_6b1c_4251_8f6c_049060957377.slice/crio-ac01a989147126215712fcf6f4d06f72eb93958b34a2084c72913b403e729e12 WatchSource:0}: Error finding container ac01a989147126215712fcf6f4d06f72eb93958b34a2084c72913b403e729e12: Status 404 returned error can't find the container with id ac01a989147126215712fcf6f4d06f72eb93958b34a2084c72913b403e729e12 Sep 11 12:31:09.310793 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.310763 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:09.310886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.310821 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:09.310886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.310851 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:09.510051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.510000 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerStarted","Data":"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478"} Sep 11 12:31:09.510246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.510073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerStarted","Data":"ac01a989147126215712fcf6f4d06f72eb93958b34a2084c72913b403e729e12"} Sep 11 12:31:09.542722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:09.542669 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podStartSLOduration=1.542653709 podStartE2EDuration="1.542653709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:09.539583254 +0000 UTC m=+1009.027788310" watchObservedRunningTime="2026-09-11 12:31:09.542653709 +0000 UTC m=+1009.030858765" Sep 11 12:31:10.030061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:10.030025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:10.035632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:10.035592 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:10.516927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:10.516892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:10.517930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:10.517909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:13.072791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.072760 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:31:13.117540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117454 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117493 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117525 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117550 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117678 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117743 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117781 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.117833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117800 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:31:13.118085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117812 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:31:13.118085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117821 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.118085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117889 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j86cl\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl\") pod \"801b682c-a4d3-44c8-9451-8921973d67ce\" (UID: \"801b682c-a4d3-44c8-9451-8921973d67ce\") " Sep 11 12:31:13.118085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.117996 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:31:13.118085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118039 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:31:13.118332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118192 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:31:13.118332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118296 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.118332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118316 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.118332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118330 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801b682c-a4d3-44c8-9451-8921973d67ce-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.118540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118343 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.118540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.118356 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.120180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.120154 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:31:13.120443 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.120423 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:31:13.120657 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.120586 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl" (OuterVolumeSpecName: "kube-api-access-j86cl") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "kube-api-access-j86cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:31:13.120657 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.120591 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "801b682c-a4d3-44c8-9451-8921973d67ce" (UID: "801b682c-a4d3-44c8-9451-8921973d67ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:31:13.219203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.219123 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.219203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.219154 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801b682c-a4d3-44c8-9451-8921973d67ce-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.219203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.219166 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801b682c-a4d3-44c8-9451-8921973d67ce-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.219203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.219178 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86cl\" (UniqueName: \"kubernetes.io/projected/801b682c-a4d3-44c8-9451-8921973d67ce-kube-api-access-j86cl\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:31:13.538698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.538666 2575 generic.go:356] "Generic (PLEG): container finished" podID="801b682c-a4d3-44c8-9451-8921973d67ce" containerID="a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f" exitCode=0 Sep 11 12:31:13.538871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.538715 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" Sep 11 12:31:13.538871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.538753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerDied","Data":"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f"} Sep 11 12:31:13.538871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.538806 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4" event={"ID":"801b682c-a4d3-44c8-9451-8921973d67ce","Type":"ContainerDied","Data":"d1a566e45be688eac9e81db0e04e3e44f6e355e257dbf71c79fb7a2235abbcf6"} Sep 11 12:31:13.538871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.538827 2575 scope.go:117] "RemoveContainer" containerID="a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f" Sep 11 12:31:13.549218 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.549200 2575 scope.go:117] "RemoveContainer" containerID="84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b" Sep 11 12:31:13.557817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.557797 2575 scope.go:117] "RemoveContainer" containerID="a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f" Sep 11 12:31:13.558199 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:31:13.558173 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f\": container with ID starting with a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f not found: ID does not exist" containerID="a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f" Sep 11 12:31:13.558199 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.558211 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f"} err="failed to get container status \"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f\": rpc error: code = NotFound desc = could not find container \"a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f\": container with ID starting with a3cfb4506085d855045d9ebc537e680f388a2799cd5bd165ba78d90b9266357f not found: ID does not exist" Sep 11 12:31:13.558488 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.558233 2575 scope.go:117] "RemoveContainer" containerID="84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b" Sep 11 12:31:13.559073 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:31:13.559051 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b\": container with ID starting with 84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b not found: ID does not exist" containerID="84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b" Sep 11 12:31:13.559152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.559078 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b"} err="failed to get container status \"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b\": rpc error: code = NotFound desc = could not find container \"84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b\": container with ID starting with 84bb2cc9db127cae0af71b3dfc3d29da4ffb94c628b139777667940ecbe5b92b not found: ID does not exist" Sep 11 12:31:13.561739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.561719 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:31:13.567195 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:13.567176 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9226c5-openshift-default-5b749b4db4-47kf4"] Sep 11 12:31:14.953679 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:14.953646 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" path="/var/lib/kubelet/pods/801b682c-a4d3-44c8-9451-8921973d67ce/volumes" Sep 11 12:31:35.696317 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.696282 2575 generic.go:356] "Generic (PLEG): container finished" podID="98ec284f-6b1c-4251-8f6c-049060957377" containerID="54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478" exitCode=0 Sep 11 12:31:35.696837 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.696361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerDied","Data":"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478"} Sep 11 12:31:35.696908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.696892 2575 scope.go:117] "RemoveContainer" containerID="54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478" Sep 11 12:31:35.698023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.697997 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:35.698088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.698063 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:35.698124 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:35.698096 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:31:36.707419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:36.707386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerStarted","Data":"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761"} Sep 11 12:31:37.030464 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:37.030380 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:37.031417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:37.031392 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 12:31:37.031529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:37.031445 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 12:31:38.030414 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:38.030378 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 12:31:38.030880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:38.030434 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 12:31:39.031118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.031080 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 12:31:39.031599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.031142 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 12:31:39.511740 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.511706 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:31:39.512367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512351 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.512424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512372 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.512424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512407 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.512424 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512415 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.512521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512506 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.512521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512517 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4779c8a4-a78d-44a3-b6d0-64c1ebb93206" containerName="istio-proxy" Sep 11 12:31:39.512586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.512530 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="801b682c-a4d3-44c8-9451-8921973d67ce" containerName="istio-proxy" Sep 11 12:31:39.516229 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.516207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.526448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.526420 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"premium-w4-cb20c8-kserve-self-signed-certs\"" Sep 11 12:31:39.526603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.526428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"openshift-service-ca.crt\"" Sep 11 12:31:39.526603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.526573 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"default-dockercfg-v4trq\"" Sep 11 12:31:39.526603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.526436 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"kube-root-ca.crt\"" Sep 11 12:31:39.532624 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.532583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:31:39.669672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.669856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.669856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.669943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjtth\" (UniqueName: \"kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.669943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.670021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.669957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770675 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.770986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.770855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjtth\" (UniqueName: \"kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.771202 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.771162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.771276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.771176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.771276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.771227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.773566 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.773517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.774255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.774234 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.780466 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.780434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjtth\" (UniqueName: \"kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth\") pod \"premium-w4-cb20c8-kserve-bc9bf9696-74sd7\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:39.836632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:39.836578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:40.030467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.030381 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 12:31:40.030467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.030453 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 12:31:40.137822 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:31:40.137794 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd577715f_55eb_46c1_a837_ed0ddd55d0c5.slice/crio-615d14393bb093c6af66eadd50f11cc107dcd0719c3e3bce1354097355ccb2d7 WatchSource:0}: Error finding container 615d14393bb093c6af66eadd50f11cc107dcd0719c3e3bce1354097355ccb2d7: Status 404 returned error can't find the container with id 615d14393bb093c6af66eadd50f11cc107dcd0719c3e3bce1354097355ccb2d7 Sep 11 12:31:40.139395 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.139366 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:31:40.517385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.517352 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:40.735296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.735254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" event={"ID":"d577715f-55eb-46c1-a837-ed0ddd55d0c5","Type":"ContainerStarted","Data":"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24"} Sep 11 12:31:40.735296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.735301 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" event={"ID":"d577715f-55eb-46c1-a837-ed0ddd55d0c5","Type":"ContainerStarted","Data":"615d14393bb093c6af66eadd50f11cc107dcd0719c3e3bce1354097355ccb2d7"} Sep 11 12:31:40.753860 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:40.753811 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" podStartSLOduration=1.753797467 podStartE2EDuration="1.753797467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:40.75347706 +0000 UTC m=+1040.241682117" watchObservedRunningTime="2026-09-11 12:31:40.753797467 +0000 UTC m=+1040.242002523" Sep 11 12:31:41.030253 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:41.030217 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Sep 11 12:31:41.030425 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:41.030271 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Sep 11 12:31:42.034707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:42.034678 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:42.035884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:42.035861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:31:49.837325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:49.837280 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:49.837781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:49.837343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:49.849114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:49.849089 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:50.514850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.514802 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:31:50.518206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.518190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.522188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.522158 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-edffad\"/\"kube-root-ca.crt\"" Sep 11 12:31:50.522188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.522165 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-edffad\"/\"default-dockercfg-nh4xl\"" Sep 11 12:31:50.522427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.522408 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-edffad\"/\"test-model-edffad-kserve-self-signed-certs\"" Sep 11 12:31:50.522505 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.522408 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-edffad\"/\"openshift-service-ca.crt\"" Sep 11 12:31:50.530907 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.530881 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:31:50.589089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4ktr\" (UniqueName: \"kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.589369 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589345 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.589539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.589681 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.589777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.589777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.589749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4ktr\" (UniqueName: \"kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690406 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690700 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690700 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690700 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690700 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.690993 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.690966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.691043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.691018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.691081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.691066 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.693065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.693039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.693310 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.693292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.702501 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.702475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4ktr\" (UniqueName: \"kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr\") pod \"test-model-edffad-kserve-756b456f8f-b5wr5\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:50.810253 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.810158 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:31:50.829404 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:50.829371 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:31:51.116956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:51.116929 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:31:51.119117 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:31:51.119097 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdf77353_38f2_4222_984f_972a53237066.slice/crio-53a65a8e73328ceb6dca54b04489e11dc759bd55a245afbea0c89a6ebaa1592c WatchSource:0}: Error finding container 53a65a8e73328ceb6dca54b04489e11dc759bd55a245afbea0c89a6ebaa1592c: Status 404 returned error can't find the container with id 53a65a8e73328ceb6dca54b04489e11dc759bd55a245afbea0c89a6ebaa1592c Sep 11 12:31:51.806779 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:51.806742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" event={"ID":"fdf77353-38f2-4222-984f-972a53237066","Type":"ContainerStarted","Data":"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6"} Sep 11 12:31:51.806779 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:51.806786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" event={"ID":"fdf77353-38f2-4222-984f-972a53237066","Type":"ContainerStarted","Data":"53a65a8e73328ceb6dca54b04489e11dc759bd55a245afbea0c89a6ebaa1592c"} Sep 11 12:31:51.828525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:31:51.828468 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" podStartSLOduration=1.8284529250000001 podStartE2EDuration="1.828452925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:31:51.826456889 +0000 UTC m=+1051.314661945" watchObservedRunningTime="2026-09-11 12:31:51.828452925 +0000 UTC m=+1051.316658015" Sep 11 12:32:00.830178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:00.830146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:32:00.830665 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:00.830220 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:32:00.842689 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:00.842670 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:32:00.873781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:00.873761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:32:15.428895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.428816 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:32:15.433897 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.433877 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.436312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.436291 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ad618118-openshift-default-dockercfg-fjvpf\"" Sep 11 12:32:15.445236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.445213 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:32:15.447300 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447402 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447587 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447731 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6pql\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.447895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.447817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6pql\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548687 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548742 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548785 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548920 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.548920 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.548890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.549156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.549138 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.549240 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.549221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.549282 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.549245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.550964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.550938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.551249 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.551231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.556741 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.556719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.557001 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.556983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6pql\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql\") pod \"e2e-mt-ad618118-openshift-default-84c8557576-n42j8\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:15.746002 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:15.745923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:16.103972 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:32:16.103940 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4de00c1_1a99_462e_8623_dcd0d77fbbe4.slice/crio-aede8609415f76abef8b014a8fe5a0e96112097df3ced1b604e2e2ee0ffa5eba WatchSource:0}: Error finding container aede8609415f76abef8b014a8fe5a0e96112097df3ced1b604e2e2ee0ffa5eba: Status 404 returned error can't find the container with id aede8609415f76abef8b014a8fe5a0e96112097df3ced1b604e2e2ee0ffa5eba Sep 11 12:32:16.106306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.106265 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:16.106387 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.106361 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:16.106461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.106444 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:16.179779 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.179753 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:32:16.972680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.972640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerStarted","Data":"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83"} Sep 11 12:32:16.973060 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:16.972706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerStarted","Data":"aede8609415f76abef8b014a8fe5a0e96112097df3ced1b604e2e2ee0ffa5eba"} Sep 11 12:32:17.002107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:17.002050 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podStartSLOduration=2.002031953 podStartE2EDuration="2.002031953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:16.999229862 +0000 UTC m=+1076.487434918" watchObservedRunningTime="2026-09-11 12:32:17.002031953 +0000 UTC m=+1076.490237010" Sep 11 12:32:17.747079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:17.747030 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:17.752873 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:17.752845 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:17.979769 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:17.979734 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:17.980667 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:17.980645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:20.820422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.820390 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk"] Sep 11 12:32:20.824154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.824132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:20.826936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.826909 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:32:20.827055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.826910 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:32:20.834534 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.834510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk"] Sep 11 12:32:20.910094 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:20.910071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk\" (UID: \"6a7bba6b-fa73-4caa-8210-6dd8d60518bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:21.010868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.010842 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk\" (UID: \"6a7bba6b-fa73-4caa-8210-6dd8d60518bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:21.013403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.013386 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:32:21.021868 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.021844 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk\" (UID: \"6a7bba6b-fa73-4caa-8210-6dd8d60518bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:21.139699 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.139673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:32:21.147917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.147899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:21.421154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:21.421116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk"] Sep 11 12:32:22.008666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:22.008627 2575 generic.go:356] "Generic (PLEG): container finished" podID="6a7bba6b-fa73-4caa-8210-6dd8d60518bc" containerID="111cb515263ce8797ed6ef8260e61b75dcc6c2ee4b0651742706ca1d69b4080e" exitCode=0 Sep 11 12:32:22.009067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:22.008713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" event={"ID":"6a7bba6b-fa73-4caa-8210-6dd8d60518bc","Type":"ContainerDied","Data":"111cb515263ce8797ed6ef8260e61b75dcc6c2ee4b0651742706ca1d69b4080e"} Sep 11 12:32:22.009067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:22.008763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" event={"ID":"6a7bba6b-fa73-4caa-8210-6dd8d60518bc","Type":"ContainerStarted","Data":"3834b483af1c5782da014dbfc19840d95e316ef0fec4fc9f0bcfcdd35653c189"} Sep 11 12:32:23.164001 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:23.163971 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:23.232716 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:23.232682 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca\") pod \"6a7bba6b-fa73-4caa-8210-6dd8d60518bc\" (UID: \"6a7bba6b-fa73-4caa-8210-6dd8d60518bc\") " Sep 11 12:32:23.233057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:23.233032 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a7bba6b-fa73-4caa-8210-6dd8d60518bc" (UID: "6a7bba6b-fa73-4caa-8210-6dd8d60518bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:32:23.334628 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:23.334523 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7bba6b-fa73-4caa-8210-6dd8d60518bc-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:24.025215 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:24.025182 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" Sep 11 12:32:24.025411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:24.025185 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk" event={"ID":"6a7bba6b-fa73-4caa-8210-6dd8d60518bc","Type":"ContainerDied","Data":"3834b483af1c5782da014dbfc19840d95e316ef0fec4fc9f0bcfcdd35653c189"} Sep 11 12:32:24.025411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:24.025312 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3834b483af1c5782da014dbfc19840d95e316ef0fec4fc9f0bcfcdd35653c189" Sep 11 12:32:31.744019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:31.743980 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk"] Sep 11 12:32:31.746800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:31.746774 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82ead5-g6qmk"] Sep 11 12:32:32.955034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:32.954997 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7bba6b-fa73-4caa-8210-6dd8d60518bc" path="/var/lib/kubelet/pods/6a7bba6b-fa73-4caa-8210-6dd8d60518bc/volumes" Sep 11 12:32:38.460701 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:38.460661 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:32:38.461089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:38.460940 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" containerID="cri-o://8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761" gracePeriod=30 Sep 11 12:32:39.208510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.208476 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:32:39.209170 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.209154 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7bba6b-fa73-4caa-8210-6dd8d60518bc" containerName="revoke-keys" Sep 11 12:32:39.209219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.209175 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7bba6b-fa73-4caa-8210-6dd8d60518bc" containerName="revoke-keys" Sep 11 12:32:39.209284 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.209273 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7bba6b-fa73-4caa-8210-6dd8d60518bc" containerName="revoke-keys" Sep 11 12:32:39.214247 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.214178 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.217330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.217303 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e961b4-openshift-default-dockercfg-nxjgk\"" Sep 11 12:32:39.219100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.219077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:32:39.315223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.315184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.315532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.315509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.315756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.315736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.315939 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.315925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.316089 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.316074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.316294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.316277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.316453 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.316435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.316652 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.316636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hst68\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.316837 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.316822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hst68\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.417968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.417947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418659 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418605 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.418746 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.418728 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.420175 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.420149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.420314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.420297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.428105 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.428081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hst68\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.428217 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.428198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token\") pod \"e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.528878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.528784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:39.815829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.815784 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:32:39.816787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.816751 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:39.816893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.816826 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:39.816893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:39.816875 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:40.141523 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:40.141491 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" event={"ID":"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d","Type":"ContainerStarted","Data":"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287"} Sep 11 12:32:40.141718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:40.141540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" event={"ID":"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d","Type":"ContainerStarted","Data":"d4d6eccd16ddec88be02bc9054adc71ab16e9584360bb86472624cb921896c33"} Sep 11 12:32:40.162063 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:40.162012 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" podStartSLOduration=1.161998562 podStartE2EDuration="1.161998562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:40.159299091 +0000 UTC m=+1099.647504161" watchObservedRunningTime="2026-09-11 12:32:40.161998562 +0000 UTC m=+1099.650203617" Sep 11 12:32:40.529738 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:40.529659 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:40.535051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:40.535025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:41.148904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:41.148869 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:41.149863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:41.149842 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:32:42.037326 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:42.037286 2575 patch_prober.go:28] interesting pod/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:32:42.037706 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:42.037350 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:32:43.743069 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.743046 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:32:43.762219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762232 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762291 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8fk4\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762313 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762375 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762396 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762418 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy\") pod \"98ec284f-6b1c-4251-8f6c-049060957377\" (UID: \"98ec284f-6b1c-4251-8f6c-049060957377\") " Sep 11 12:32:43.762818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:32:43.762818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762669 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:32:43.763023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762882 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data" (OuterVolumeSpecName: "istio-data") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:32:43.763023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762927 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:32:43.763023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.762943 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:32:43.763936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.763909 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.764051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.763938 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.764051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.763965 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.764051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.763980 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.764051 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.763993 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98ec284f-6b1c-4251-8f6c-049060957377-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.765148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.765131 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:32:43.765333 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.765265 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4" (OuterVolumeSpecName: "kube-api-access-f8fk4") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "kube-api-access-f8fk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:32:43.765925 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.765536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:32:43.765925 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.765743 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token" (OuterVolumeSpecName: "istio-token") pod "98ec284f-6b1c-4251-8f6c-049060957377" (UID: "98ec284f-6b1c-4251-8f6c-049060957377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:32:43.864869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.864847 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8fk4\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-kube-api-access-f8fk4\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.864869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.864867 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98ec284f-6b1c-4251-8f6c-049060957377-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.864988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.864876 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98ec284f-6b1c-4251-8f6c-049060957377-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:43.864988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:43.864885 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98ec284f-6b1c-4251-8f6c-049060957377-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:44.169748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.169719 2575 generic.go:356] "Generic (PLEG): container finished" podID="98ec284f-6b1c-4251-8f6c-049060957377" containerID="8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761" exitCode=0 Sep 11 12:32:44.169748 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.169753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerDied","Data":"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761"} Sep 11 12:32:44.170008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.169790 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" Sep 11 12:32:44.170008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.169814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg" event={"ID":"98ec284f-6b1c-4251-8f6c-049060957377","Type":"ContainerDied","Data":"ac01a989147126215712fcf6f4d06f72eb93958b34a2084c72913b403e729e12"} Sep 11 12:32:44.170008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.169834 2575 scope.go:117] "RemoveContainer" containerID="8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761" Sep 11 12:32:44.181859 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.181830 2575 scope.go:117] "RemoveContainer" containerID="54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478" Sep 11 12:32:44.193301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.193280 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:32:44.193657 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.193601 2575 scope.go:117] "RemoveContainer" containerID="8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761" Sep 11 12:32:44.193923 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:32:44.193901 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761\": container with ID starting with 8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761 not found: ID does not exist" containerID="8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761" Sep 11 12:32:44.194007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.193929 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761"} err="failed to get container status \"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761\": rpc error: code = NotFound desc = could not find container \"8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761\": container with ID starting with 8083a8cfad2558a9947ce9f641b09eca7e2989147200c80227107b44283e6761 not found: ID does not exist" Sep 11 12:32:44.194007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.193947 2575 scope.go:117] "RemoveContainer" containerID="54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478" Sep 11 12:32:44.194207 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:32:44.194193 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478\": container with ID starting with 54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478 not found: ID does not exist" containerID="54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478" Sep 11 12:32:44.194248 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.194212 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478"} err="failed to get container status \"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478\": rpc error: code = NotFound desc = could not find container \"54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478\": container with ID starting with 54e53d6e498bda29893c869637024489fc88b20fe535be2ca8041af40881e478 not found: ID does not exist" Sep 11 12:32:44.196358 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.196341 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-82ead5-openshift-default-67c5556c5b-btkxg"] Sep 11 12:32:44.955181 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:44.955146 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ec284f-6b1c-4251-8f6c-049060957377" path="/var/lib/kubelet/pods/98ec284f-6b1c-4251-8f6c-049060957377/volumes" Sep 11 12:32:54.203579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.203550 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9"] Sep 11 12:32:54.204110 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.204094 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:32:54.204151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.204112 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:32:54.204208 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.204200 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:32:54.204247 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.204210 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:32:54.207140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.207117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:54.210131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.210082 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:32:54.210131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.210102 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:32:54.212968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.212945 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9"] Sep 11 12:32:54.251311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.251283 2575 generic.go:356] "Generic (PLEG): container finished" podID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerID="c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83" exitCode=0 Sep 11 12:32:54.251451 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.251350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerDied","Data":"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83"} Sep 11 12:32:54.251811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.251791 2575 scope.go:117] "RemoveContainer" containerID="c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83" Sep 11 12:32:54.253047 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.252981 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:54.253148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.253049 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:54.253148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.253080 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:32:54.282972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.282929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad618118-n7jt9\" (UID: \"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:54.384637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.384520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad618118-n7jt9\" (UID: \"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:54.385224 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.385200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad618118-n7jt9\" (UID: \"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:54.550789 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.550709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:54.839886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:54.839602 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9"] Sep 11 12:32:55.259352 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.259315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerStarted","Data":"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1"} Sep 11 12:32:55.261071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.261042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" event={"ID":"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6","Type":"ContainerStarted","Data":"46faa3174600e736ea216e16c56bfc650e5b1548bc10ca2ed19a8f09ae29fb38"} Sep 11 12:32:55.261174 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.261093 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" event={"ID":"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6","Type":"ContainerStarted","Data":"f2cff6d030e178b088dba17ba6ce40e51e45b8685e8c76d30887bca9799a4205"} Sep 11 12:32:55.300389 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.300328 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" podStartSLOduration=1.300309084 podStartE2EDuration="1.300309084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:32:55.299018395 +0000 UTC m=+1114.787223455" watchObservedRunningTime="2026-09-11 12:32:55.300309084 +0000 UTC m=+1114.788514142" Sep 11 12:32:55.746993 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.746959 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:32:55.747186 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.747154 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 12:32:55.747234 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:55.747202 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 12:32:56.269141 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:56.269108 2575 generic.go:356] "Generic (PLEG): container finished" podID="cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" containerID="46faa3174600e736ea216e16c56bfc650e5b1548bc10ca2ed19a8f09ae29fb38" exitCode=0 Sep 11 12:32:56.269532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:56.269203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" event={"ID":"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6","Type":"ContainerDied","Data":"46faa3174600e736ea216e16c56bfc650e5b1548bc10ca2ed19a8f09ae29fb38"} Sep 11 12:32:56.747064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:56.747024 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 12:32:56.747259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:56.747084 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 12:32:57.461762 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.461735 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:57.523556 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.523523 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca\") pod \"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6\" (UID: \"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6\") " Sep 11 12:32:57.523901 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.523875 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" (UID: "cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:32:57.625247 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.625163 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:32:57.746571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.746531 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 12:32:57.746790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:57.746602 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 12:32:58.292123 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:58.292079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" event={"ID":"cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6","Type":"ContainerDied","Data":"f2cff6d030e178b088dba17ba6ce40e51e45b8685e8c76d30887bca9799a4205"} Sep 11 12:32:58.292333 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:58.292146 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2cff6d030e178b088dba17ba6ce40e51e45b8685e8c76d30887bca9799a4205" Sep 11 12:32:58.292333 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:58.292210 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9" Sep 11 12:32:58.746541 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:58.746498 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 12:32:58.746990 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:58.746571 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 12:32:59.746711 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:59.746675 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 12:32:59.747021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:32:59.746736 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 12:33:00.751359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:00.751324 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:33:00.751834 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:00.751727 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:33:00.752399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:00.752383 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:33:04.966157 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:04.966125 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9"] Sep 11 12:33:04.968979 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:04.968958 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad618118-n7jt9"] Sep 11 12:33:06.954811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:06.954773 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" path="/var/lib/kubelet/pods/cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6/volumes" Sep 11 12:33:11.414329 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:11.414291 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:33:11.414787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:11.414536 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" containerID="cri-o://9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1" gracePeriod=30 Sep 11 12:33:15.754642 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:15.754586 2575 patch_prober.go:28] interesting pod/e2e-mt-ad618118-openshift-default-84c8557576-n42j8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:33:15.755020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:15.754675 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:33:16.689332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.689264 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:33:16.735711 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735666 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.735908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735739 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.735908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735771 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.735908 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735880 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735936 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735966 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.735997 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6pql\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736018 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:16.736088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736030 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:16.736351 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736351 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736149 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy\") pod \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\" (UID: \"d4de00c1-1a99-462e-8623-dcd0d77fbbe4\") " Sep 11 12:33:16.736351 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736280 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:16.736677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736586 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.736677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736635 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.736677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736651 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.736677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736584 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:33:16.736677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.736604 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:16.738493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.738460 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql" (OuterVolumeSpecName: "kube-api-access-l6pql") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "kube-api-access-l6pql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:33:16.738493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.738473 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:33:16.738857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.738835 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:16.739061 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.739027 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4de00c1-1a99-462e-8623-dcd0d77fbbe4" (UID: "d4de00c1-1a99-462e-8623-dcd0d77fbbe4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:33:16.837938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837898 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.837938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837931 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.837938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837940 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6pql\" (UniqueName: \"kubernetes.io/projected/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-kube-api-access-l6pql\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.838390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837951 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.838390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837960 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:16.838390 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:16.837967 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4de00c1-1a99-462e-8623-dcd0d77fbbe4-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:17.446726 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.446691 2575 generic.go:356] "Generic (PLEG): container finished" podID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerID="9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1" exitCode=0 Sep 11 12:33:17.446902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.446737 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerDied","Data":"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1"} Sep 11 12:33:17.446902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.446780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" event={"ID":"d4de00c1-1a99-462e-8623-dcd0d77fbbe4","Type":"ContainerDied","Data":"aede8609415f76abef8b014a8fe5a0e96112097df3ced1b604e2e2ee0ffa5eba"} Sep 11 12:33:17.446902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.446801 2575 scope.go:117] "RemoveContainer" containerID="9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1" Sep 11 12:33:17.446902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.446740 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8" Sep 11 12:33:17.456940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.456924 2575 scope.go:117] "RemoveContainer" containerID="c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83" Sep 11 12:33:17.466339 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.466312 2575 scope.go:117] "RemoveContainer" containerID="9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1" Sep 11 12:33:17.466899 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:33:17.466665 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1\": container with ID starting with 9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1 not found: ID does not exist" containerID="9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1" Sep 11 12:33:17.466899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.466709 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1"} err="failed to get container status \"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1\": rpc error: code = NotFound desc = could not find container \"9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1\": container with ID starting with 9a49ec7fcc9477034aab2fb3de0dc60b1e8bf8e36046247d06177ef7b66704e1 not found: ID does not exist" Sep 11 12:33:17.466899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.466737 2575 scope.go:117] "RemoveContainer" containerID="c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83" Sep 11 12:33:17.467485 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:33:17.467457 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83\": container with ID starting with c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83 not found: ID does not exist" containerID="c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83" Sep 11 12:33:17.467596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.467496 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83"} err="failed to get container status \"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83\": rpc error: code = NotFound desc = could not find container \"c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83\": container with ID starting with c2956676df0d06d65fc4a2d1a1664671312efa7353ae879ff7081e100e0e0b83 not found: ID does not exist" Sep 11 12:33:17.467693 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.467603 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:33:17.470586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:17.470561 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ad618118-openshift-default-84c8557576-n42j8"] Sep 11 12:33:18.956682 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:18.956645 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" path="/var/lib/kubelet/pods/d4de00c1-1a99-462e-8623-dcd0d77fbbe4/volumes" Sep 11 12:33:24.196185 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:24.196151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:33:24.196738 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:24.196414 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerName="istio-proxy" containerID="cri-o://5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287" gracePeriod=30 Sep 11 12:33:25.499427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499388 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:33:25.499957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499960 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ec284f-6b1c-4251-8f6c-049060957377" containerName="istio-proxy" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499982 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499987 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.499995 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" containerName="revoke-keys" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.500000 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" containerName="revoke-keys" Sep 11 12:33:25.500016 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.500014 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:25.500223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.500019 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:25.500223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.500083 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:25.500223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.500094 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc3bbe40-2c26-48fc-a2b4-1bab2ac3a5a6" containerName="revoke-keys" Sep 11 12:33:25.504814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.504794 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.507160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.507141 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-44f372-openshift-default-dockercfg-ddh7b\"" Sep 11 12:33:25.519681 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.519652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:33:25.625511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625641 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nccd8\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.625884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.625870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.726992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccd8\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727525 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.727937 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.727908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.729043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.729009 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.729167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.729147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.729278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.729257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.729601 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.729578 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.731409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.731387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.734100 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.732588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.737894 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.737849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.739220 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.739198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccd8\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8\") pod \"e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:25.818117 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:25.818053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:26.104792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.104754 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:33:26.105975 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:33:26.105948 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda767dd78_1d06_4cd8_b86d_575206369ea0.slice/crio-b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5 WatchSource:0}: Error finding container b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5: Status 404 returned error can't find the container with id b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5 Sep 11 12:33:26.108359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.108329 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:33:26.108431 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.108392 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:33:26.108431 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.108419 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:33:26.150948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.150920 2575 patch_prober.go:28] interesting pod/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:33:26.151053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.150977 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:33:26.510214 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.510171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerStarted","Data":"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b"} Sep 11 12:33:26.510722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.510241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerStarted","Data":"b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5"} Sep 11 12:33:26.533535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.533487 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podStartSLOduration=1.533470306 podStartE2EDuration="1.533470306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:26.532168494 +0000 UTC m=+1146.020373551" watchObservedRunningTime="2026-09-11 12:33:26.533470306 +0000 UTC m=+1146.021675427" Sep 11 12:33:26.818786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:26.818683 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:27.823973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:27.823943 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:28.522869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:28.522840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:28.523830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:28.523810 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:33:29.448227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.448201 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:33:29.532035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.531938 2575 generic.go:356] "Generic (PLEG): container finished" podID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerID="5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287" exitCode=0 Sep 11 12:33:29.532035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.531983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" event={"ID":"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d","Type":"ContainerDied","Data":"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287"} Sep 11 12:33:29.532035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.531999 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" Sep 11 12:33:29.532325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.532044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l" event={"ID":"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d","Type":"ContainerDied","Data":"d4d6eccd16ddec88be02bc9054adc71ab16e9584360bb86472624cb921896c33"} Sep 11 12:33:29.532325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.532072 2575 scope.go:117] "RemoveContainer" containerID="5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287" Sep 11 12:33:29.546410 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.546385 2575 scope.go:117] "RemoveContainer" containerID="5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287" Sep 11 12:33:29.546755 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:33:29.546735 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287\": container with ID starting with 5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287 not found: ID does not exist" containerID="5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287" Sep 11 12:33:29.546815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.546766 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287"} err="failed to get container status \"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287\": rpc error: code = NotFound desc = could not find container \"5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287\": container with ID starting with 5221a1c8c6aad79aa44f374aac05ba5fa22ffe6301084f592b5636e420cbe287 not found: ID does not exist" Sep 11 12:33:29.568139 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568112 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568245 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hst68\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568245 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568259 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568294 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568318 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568366 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568455 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket\") pod \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\" (UID: \"458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d\") " Sep 11 12:33:29.568718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:29.568718 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568701 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:29.568979 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568913 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:29.568979 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.568929 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:29.569159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569018 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:33:29.569480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569457 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.569561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569485 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.569561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569500 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.569561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569514 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.569561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.569527 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.570492 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.570465 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:29.570639 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.570523 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68" (OuterVolumeSpecName: "kube-api-access-hst68") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "kube-api-access-hst68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:33:29.570639 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.570578 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:33:29.570954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.570927 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" (UID: "458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:33:29.671153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.671104 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.671153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.671151 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.671381 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.671165 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hst68\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-kube-api-access-hst68\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.671381 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.671182 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:29.860367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.860331 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:33:29.864027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:29.863999 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e961b4-openshift-default-854c8cbb8b-mkn9l"] Sep 11 12:33:30.955467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:30.955436 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" path="/var/lib/kubelet/pods/458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d/volumes" Sep 11 12:33:36.680799 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.680753 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:33:36.681315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.681207 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerName="istio-proxy" Sep 11 12:33:36.681315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.681219 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerName="istio-proxy" Sep 11 12:33:36.681315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.681285 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="458e85b9-8bdf-43ef-b7db-dc9ff0c5be9d" containerName="istio-proxy" Sep 11 12:33:36.681315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.681291 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4de00c1-1a99-462e-8623-dcd0d77fbbe4" containerName="istio-proxy" Sep 11 12:33:36.686261 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.686239 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.689827 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.689799 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"e2e-conflict-b-97e93ffc-kserve-self-signed-certs\"" Sep 11 12:33:36.693159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.693134 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:33:36.848881 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.848842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.849088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.848927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.849088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.848963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.849088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.849002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.849088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.849044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnm6\" (UniqueName: \"kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.849292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.849157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.949797 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.949797 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnm6\" (UniqueName: \"kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.949934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.950198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.950321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.950385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.950366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.952318 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.952297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.952707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.952686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.959296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.959271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnm6\" (UniqueName: \"kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6\") pod \"e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:36.996817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:36.996767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:37.275951 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:37.275921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:33:37.277380 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:33:37.277350 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod358bb651_bd82_462a_878d_e23ff344e3c0.slice/crio-7a9285a3867d37cce748b6d5e6c237e8edc2c9af4db719e1f76f4de3319a9de8 WatchSource:0}: Error finding container 7a9285a3867d37cce748b6d5e6c237e8edc2c9af4db719e1f76f4de3319a9de8: Status 404 returned error can't find the container with id 7a9285a3867d37cce748b6d5e6c237e8edc2c9af4db719e1f76f4de3319a9de8 Sep 11 12:33:37.592125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:37.592030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" event={"ID":"358bb651-bd82-462a-878d-e23ff344e3c0","Type":"ContainerStarted","Data":"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833"} Sep 11 12:33:37.592125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:37.592081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" event={"ID":"358bb651-bd82-462a-878d-e23ff344e3c0","Type":"ContainerStarted","Data":"7a9285a3867d37cce748b6d5e6c237e8edc2c9af4db719e1f76f4de3319a9de8"} Sep 11 12:33:37.611494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:37.611446 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" podStartSLOduration=1.611431477 podStartE2EDuration="1.611431477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:37.610694641 +0000 UTC m=+1157.098899699" watchObservedRunningTime="2026-09-11 12:33:37.611431477 +0000 UTC m=+1157.099636534" Sep 11 12:33:41.379938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.379899 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:33:41.495364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.495302 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj"] Sep 11 12:33:41.498882 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.498855 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:41.501873 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.501724 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:33:41.501873 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.501759 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:33:41.505020 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.504984 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj"] Sep 11 12:33:41.602138 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.602082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj\" (UID: \"5a777029-3f9c-4887-bafe-f519cd92c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:41.703949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.703911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj\" (UID: \"5a777029-3f9c-4887-bafe-f519cd92c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:41.704577 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.704556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj\" (UID: \"5a777029-3f9c-4887-bafe-f519cd92c1df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:41.813369 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:41.813326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:42.095376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:42.095347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj"] Sep 11 12:33:42.096184 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:33:42.096162 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a777029_3f9c_4887_bafe_f519cd92c1df.slice/crio-e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff WatchSource:0}: Error finding container e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff: Status 404 returned error can't find the container with id e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff Sep 11 12:33:42.634422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:42.634325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" event={"ID":"5a777029-3f9c-4887-bafe-f519cd92c1df","Type":"ContainerStarted","Data":"b59e31065386f2b575860025103c59bde11b8ae679947994702d5e8a122ffff1"} Sep 11 12:33:42.634422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:42.634380 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" event={"ID":"5a777029-3f9c-4887-bafe-f519cd92c1df","Type":"ContainerStarted","Data":"e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff"} Sep 11 12:33:42.652720 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:42.652645 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" podStartSLOduration=1.652602832 podStartE2EDuration="1.652602832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:33:42.652232286 +0000 UTC m=+1162.140437343" watchObservedRunningTime="2026-09-11 12:33:42.652602832 +0000 UTC m=+1162.140807888" Sep 11 12:33:43.642863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:43.642827 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a777029-3f9c-4887-bafe-f519cd92c1df" containerID="b59e31065386f2b575860025103c59bde11b8ae679947994702d5e8a122ffff1" exitCode=0 Sep 11 12:33:43.643279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:43.642873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" event={"ID":"5a777029-3f9c-4887-bafe-f519cd92c1df","Type":"ContainerDied","Data":"b59e31065386f2b575860025103c59bde11b8ae679947994702d5e8a122ffff1"} Sep 11 12:33:44.803257 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:44.803224 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:44.838604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:44.838567 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca\") pod \"5a777029-3f9c-4887-bafe-f519cd92c1df\" (UID: \"5a777029-3f9c-4887-bafe-f519cd92c1df\") " Sep 11 12:33:44.838974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:44.838949 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a777029-3f9c-4887-bafe-f519cd92c1df" (UID: "5a777029-3f9c-4887-bafe-f519cd92c1df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:33:44.839113 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:44.839098 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a777029-3f9c-4887-bafe-f519cd92c1df-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:45.657125 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:45.657097 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" Sep 11 12:33:45.657315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:45.657139 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj" event={"ID":"5a777029-3f9c-4887-bafe-f519cd92c1df","Type":"ContainerDied","Data":"e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff"} Sep 11 12:33:45.657315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:45.657191 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e304b1c3d71ff04dbece88b89f1e1ea9a712f60457afcf918257e03a98a154ff" Sep 11 12:33:46.997695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:46.997647 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:46.998198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:46.997717 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:47.011478 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:47.011439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:47.682681 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:47.682642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:33:51.904387 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:51.904344 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj"] Sep 11 12:33:51.908544 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:51.908500 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-38a724-j6zkj"] Sep 11 12:33:52.955258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:52.955197 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a777029-3f9c-4887-bafe-f519cd92c1df" path="/var/lib/kubelet/pods/5a777029-3f9c-4887-bafe-f519cd92c1df/volumes" Sep 11 12:33:56.403850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.403780 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" podUID="fdf77353-38f2-4222-984f-972a53237066" containerName="main" containerID="cri-o://14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6" gracePeriod=45 Sep 11 12:33:56.726819 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.726781 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:33:56.759711 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.759673 2575 generic.go:356] "Generic (PLEG): container finished" podID="fdf77353-38f2-4222-984f-972a53237066" containerID="14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6" exitCode=0 Sep 11 12:33:56.759711 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.759717 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" Sep 11 12:33:56.760346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.759792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" event={"ID":"fdf77353-38f2-4222-984f-972a53237066","Type":"ContainerDied","Data":"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6"} Sep 11 12:33:56.760346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.759848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5" event={"ID":"fdf77353-38f2-4222-984f-972a53237066","Type":"ContainerDied","Data":"53a65a8e73328ceb6dca54b04489e11dc759bd55a245afbea0c89a6ebaa1592c"} Sep 11 12:33:56.760346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.759873 2575 scope.go:117] "RemoveContainer" containerID="14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6" Sep 11 12:33:56.771502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.771463 2575 scope.go:117] "RemoveContainer" containerID="14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6" Sep 11 12:33:56.794651 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:33:56.786851 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6\": container with ID starting with 14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6 not found: ID does not exist" containerID="14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6" Sep 11 12:33:56.794651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.786953 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6"} err="failed to get container status \"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6\": rpc error: code = NotFound desc = could not find container \"14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6\": container with ID starting with 14b3d2a35a1f4156e585d35bb184c190d974001a8858b3a5f60087182dba42f6 not found: ID does not exist" Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798028 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798229 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798375 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4ktr\" (UniqueName: \"kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.798401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm\") pod \"fdf77353-38f2-4222-984f-972a53237066\" (UID: \"fdf77353-38f2-4222-984f-972a53237066\") " Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.799446 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:56.802561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.801900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:56.803783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.803730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home" (OuterVolumeSpecName: "home") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:56.807584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.807494 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:33:56.807584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.807538 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr" (OuterVolumeSpecName: "kube-api-access-h4ktr") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "kube-api-access-h4ktr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:33:56.807584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.807534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm" (OuterVolumeSpecName: "dshm") pod "fdf77353-38f2-4222-984f-972a53237066" (UID: "fdf77353-38f2-4222-984f-972a53237066"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:33:56.900538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900494 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:56.900538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900541 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4ktr\" (UniqueName: \"kubernetes.io/projected/fdf77353-38f2-4222-984f-972a53237066-kube-api-access-h4ktr\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:56.900840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900559 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:56.900840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900574 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:56.900840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900588 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf77353-38f2-4222-984f-972a53237066-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:56.900840 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:56.900602 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf77353-38f2-4222-984f-972a53237066-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:33:57.078851 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:57.078801 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:33:57.084303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:57.084258 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-edffad/test-model-edffad-kserve-756b456f8f-b5wr5"] Sep 11 12:33:58.955407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:58.955358 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf77353-38f2-4222-984f-972a53237066" path="/var/lib/kubelet/pods/fdf77353-38f2-4222-984f-972a53237066/volumes" Sep 11 12:33:59.288699 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.288580 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:33:59.288916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.288877 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" containerID="cri-o://91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5" gracePeriod=30 Sep 11 12:33:59.564972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.564852 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:33:59.564972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.564944 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:33:59.652465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.652407 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb"] Sep 11 12:33:59.653195 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653164 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a777029-3f9c-4887-bafe-f519cd92c1df" containerName="revoke-keys" Sep 11 12:33:59.653276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653197 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a777029-3f9c-4887-bafe-f519cd92c1df" containerName="revoke-keys" Sep 11 12:33:59.653276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653215 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf77353-38f2-4222-984f-972a53237066" containerName="main" Sep 11 12:33:59.653276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653224 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf77353-38f2-4222-984f-972a53237066" containerName="main" Sep 11 12:33:59.653384 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653371 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf77353-38f2-4222-984f-972a53237066" containerName="main" Sep 11 12:33:59.653418 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.653395 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a777029-3f9c-4887-bafe-f519cd92c1df" containerName="revoke-keys" Sep 11 12:33:59.659190 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.659138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:33:59.661955 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.661918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:33:59.661955 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.661928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:33:59.663260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.663221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb"] Sep 11 12:33:59.734599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.734556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb\" (UID: \"93fa21da-723d-46a5-9030-bce9f75e140a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:33:59.836023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.835887 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb\" (UID: \"93fa21da-723d-46a5-9030-bce9f75e140a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:33:59.836676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.836649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb\" (UID: \"93fa21da-723d-46a5-9030-bce9f75e140a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:33:59.978782 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:33:59.978740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:34:00.272984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:00.272893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb"] Sep 11 12:34:00.274634 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:00.274569 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fa21da_723d_46a5_9030_bce9f75e140a.slice/crio-b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d WatchSource:0}: Error finding container b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d: Status 404 returned error can't find the container with id b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d Sep 11 12:34:00.807425 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:00.807387 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" event={"ID":"93fa21da-723d-46a5-9030-bce9f75e140a","Type":"ContainerStarted","Data":"eb1ab6926f4c0992bfbb30674d4816adf1cb33e735ffe9b6eb86cb57a101dbd6"} Sep 11 12:34:00.807652 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:00.807454 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" event={"ID":"93fa21da-723d-46a5-9030-bce9f75e140a","Type":"ContainerStarted","Data":"b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d"} Sep 11 12:34:00.824367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:00.824295 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" podStartSLOduration=1.8242744819999999 podStartE2EDuration="1.824274482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:00.822910781 +0000 UTC m=+1180.311115839" watchObservedRunningTime="2026-09-11 12:34:00.824274482 +0000 UTC m=+1180.312479538" Sep 11 12:34:01.820426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.820386 2575 generic.go:356] "Generic (PLEG): container finished" podID="93fa21da-723d-46a5-9030-bce9f75e140a" containerID="eb1ab6926f4c0992bfbb30674d4816adf1cb33e735ffe9b6eb86cb57a101dbd6" exitCode=0 Sep 11 12:34:01.820949 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.820476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" event={"ID":"93fa21da-723d-46a5-9030-bce9f75e140a","Type":"ContainerDied","Data":"eb1ab6926f4c0992bfbb30674d4816adf1cb33e735ffe9b6eb86cb57a101dbd6"} Sep 11 12:34:01.822501 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.822471 2575 generic.go:356] "Generic (PLEG): container finished" podID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerID="9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b" exitCode=0 Sep 11 12:34:01.822655 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.822544 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerDied","Data":"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b"} Sep 11 12:34:01.823064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.823045 2575 scope.go:117] "RemoveContainer" containerID="9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b" Sep 11 12:34:01.824440 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.824411 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:01.824504 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.824483 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:01.824540 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:01.824517 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:02.832133 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:02.832094 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerStarted","Data":"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6"} Sep 11 12:34:03.030437 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.030391 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:34:03.100562 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.100433 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca\") pod \"93fa21da-723d-46a5-9030-bce9f75e140a\" (UID: \"93fa21da-723d-46a5-9030-bce9f75e140a\") " Sep 11 12:34:03.101678 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.101255 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93fa21da-723d-46a5-9030-bce9f75e140a" (UID: "93fa21da-723d-46a5-9030-bce9f75e140a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:34:03.102220 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.102187 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93fa21da-723d-46a5-9030-bce9f75e140a-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:03.819071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.819017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:34:03.819346 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.819319 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 12:34:03.819415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.819379 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 12:34:03.841855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.841818 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" Sep 11 12:34:03.842277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.841820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb" event={"ID":"93fa21da-723d-46a5-9030-bce9f75e140a","Type":"ContainerDied","Data":"b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d"} Sep 11 12:34:03.842277 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:03.841967 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b773f199af87625db307c75b6ce051404bf863b1655c1edde4124413604d6d" Sep 11 12:34:04.587206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.586769 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:34:04.622991 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.622945 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.622991 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.622996 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623037 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv6b5\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623135 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623163 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623189 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623221 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket\") pod \"db8da46f-8e69-46d8-ae49-ab8a77d51024\" (UID: \"db8da46f-8e69-46d8-ae49-ab8a77d51024\") " Sep 11 12:34:04.623787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623392 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:04.623787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623408 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:34:04.623895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623817 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db8da46f-8e69-46d8-ae49-ab8a77d51024-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.623895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.623840 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.624409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.624356 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:04.624591 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.624547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:04.624703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.624419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data" (OuterVolumeSpecName: "istio-data") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:04.626778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.626738 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:04.626778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.626762 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:34:04.627185 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.627155 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token" (OuterVolumeSpecName: "istio-token") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:04.627270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.627242 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5" (OuterVolumeSpecName: "kube-api-access-cv6b5") pod "db8da46f-8e69-46d8-ae49-ab8a77d51024" (UID: "db8da46f-8e69-46d8-ae49-ab8a77d51024"). InnerVolumeSpecName "kube-api-access-cv6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:04.725083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725027 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725072 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6b5\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-kube-api-access-cv6b5\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725083 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725094 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725083 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725103 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725111 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.725459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.725120 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db8da46f-8e69-46d8-ae49-ab8a77d51024-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:04.819826 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.819706 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 12:34:04.819826 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.819777 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 12:34:04.852266 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.852231 2575 generic.go:356] "Generic (PLEG): container finished" podID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerID="91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5" exitCode=0 Sep 11 12:34:04.852712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.852283 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" Sep 11 12:34:04.852712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.852314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerDied","Data":"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5"} Sep 11 12:34:04.852712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.852406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f" event={"ID":"db8da46f-8e69-46d8-ae49-ab8a77d51024","Type":"ContainerDied","Data":"b7cbf642e72047a3c593811f5bac91b1a3b50f7414ab5220c8fdd8f51358f45c"} Sep 11 12:34:04.852712 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.852436 2575 scope.go:117] "RemoveContainer" containerID="91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5" Sep 11 12:34:04.865638 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.865589 2575 scope.go:117] "RemoveContainer" containerID="017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.890785 2575 scope.go:117] "RemoveContainer" containerID="91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:04.892128 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5\": container with ID starting with 91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5 not found: ID does not exist" containerID="91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.892176 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5"} err="failed to get container status \"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5\": rpc error: code = NotFound desc = could not find container \"91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5\": container with ID starting with 91b5d05999da0abfc446fd362fb489d9d142694623871e0ac459aaec2a9387f5 not found: ID does not exist" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.892207 2575 scope.go:117] "RemoveContainer" containerID="017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.892317 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:04.892730 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99\": container with ID starting with 017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99 not found: ID does not exist" containerID="017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99" Sep 11 12:34:04.893650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.892762 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99"} err="failed to get container status \"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99\": rpc error: code = NotFound desc = could not find container \"017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99\": container with ID starting with 017337dbe7cf4ed386ced404faab3da97e5c8ff2610a13d7b0d7b0d925ea8f99 not found: ID does not exist" Sep 11 12:34:04.902787 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.902735 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-38a724-openshift-default-579bf8db4-pc95f"] Sep 11 12:34:04.955588 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:04.955541 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" path="/var/lib/kubelet/pods/db8da46f-8e69-46d8-ae49-ab8a77d51024/volumes" Sep 11 12:34:05.819269 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:05.819195 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 12:34:05.819555 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:05.819386 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 12:34:06.819050 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:06.819005 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 12:34:06.819469 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:06.819109 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 12:34:07.784104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784031 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:34:07.784830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784799 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93fa21da-723d-46a5-9030-bce9f75e140a" containerName="revoke-keys" Sep 11 12:34:07.784830 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784829 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="93fa21da-723d-46a5-9030-bce9f75e140a" containerName="revoke-keys" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784845 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784855 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784886 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.784896 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.785045 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.785058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.785057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="93fa21da-723d-46a5-9030-bce9f75e140a" containerName="revoke-keys" Sep 11 12:34:07.785375 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.785070 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="db8da46f-8e69-46d8-ae49-ab8a77d51024" containerName="istio-proxy" Sep 11 12:34:07.793475 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.793437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.804137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.799335 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"default-dockercfg-kmx7t\"" Sep 11 12:34:07.804137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.799590 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"distinct-2-w0-1a8ee3-kserve-self-signed-certs\"" Sep 11 12:34:07.804137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.799833 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"kube-root-ca.crt\"" Sep 11 12:34:07.804137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.800044 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"openshift-service-ca.crt\"" Sep 11 12:34:07.804137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.804075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:34:07.818848 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.818811 2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 12:34:07.819053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.818872 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 12:34:07.859407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859372 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.859857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.859857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.859857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lssd\" (UniqueName: \"kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.859857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.859857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.859574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961506 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lssd\" (UniqueName: \"kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.961905 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.961887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.962108 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.962088 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.964022 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.963995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.966252 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.966229 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:07.970064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:07.970040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lssd\" (UniqueName: \"kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd\") pod \"distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:08.110833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.110470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:08.396072 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.396034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:34:08.396582 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:08.396547 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0bc8c1d_c77c_4c19_8a6f_fac0525ba053.slice/crio-38a4a9c638c4b342503475855586798773f49ac2f6ff3054502ed11ecfa38755 WatchSource:0}: Error finding container 38a4a9c638c4b342503475855586798773f49ac2f6ff3054502ed11ecfa38755: Status 404 returned error can't find the container with id 38a4a9c638c4b342503475855586798773f49ac2f6ff3054502ed11ecfa38755 Sep 11 12:34:08.822803 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.822714 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:34:08.823110 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.823094 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:34:08.823767 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.823749 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:34:08.901511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.901474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" event={"ID":"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053","Type":"ContainerStarted","Data":"11489f6aa5fb342912fd6d8bcf8d342058ce5970cd83743545034769e0c4922b"} Sep 11 12:34:08.901964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.901526 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" event={"ID":"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053","Type":"ContainerStarted","Data":"38a4a9c638c4b342503475855586798773f49ac2f6ff3054502ed11ecfa38755"} Sep 11 12:34:08.922904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:08.922846 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" podStartSLOduration=1.922823468 podStartE2EDuration="1.922823468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:08.919602848 +0000 UTC m=+1188.407807906" watchObservedRunningTime="2026-09-11 12:34:08.922823468 +0000 UTC m=+1188.411028524" Sep 11 12:34:10.255945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.255903 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz"] Sep 11 12:34:10.259458 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.259425 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-edffad-29818830-xpffz"] Sep 11 12:34:10.518088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.517989 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb"] Sep 11 12:34:10.523181 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.523146 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-edffad-qkkkb"] Sep 11 12:34:10.953516 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.953481 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05628a32-a10c-4217-8b7b-9632cce21f7c" path="/var/lib/kubelet/pods/05628a32-a10c-4217-8b7b-9632cce21f7c/volumes" Sep 11 12:34:10.953858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:10.953843 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93fa21da-723d-46a5-9030-bce9f75e140a" path="/var/lib/kubelet/pods/93fa21da-723d-46a5-9030-bce9f75e140a/volumes" Sep 11 12:34:17.107480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:17.107450 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:34:17.107970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:17.107674 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" containerID="cri-o://1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600" gracePeriod=30 Sep 11 12:34:18.111567 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:18.111533 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:18.111940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:18.111584 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:18.123834 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:18.123807 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:18.982179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:18.982152 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:34:20.997918 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:20.997887 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:34:20.998794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:20.998775 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:34:22.356304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.356271 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:34:22.518086 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518100 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518165 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518232 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518279 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518295 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518368 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkzvz\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518407 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket\") pod \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\" (UID: \"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394\") " Sep 11 12:34:22.518517 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518453 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518721 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518861 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518872 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518895 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518910 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.518984 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.518923 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.520487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.520456 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:34:22.520698 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.520676 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:22.521209 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.521191 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:34:22.521264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.521206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz" (OuterVolumeSpecName: "kube-api-access-tkzvz") pod "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" (UID: "b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394"). InnerVolumeSpecName "kube-api-access-tkzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:22.619637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.619574 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkzvz\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-kube-api-access-tkzvz\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.619637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.619602 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.619825 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.619663 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.619825 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.619672 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.619825 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:22.619680 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:22.989982 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:22.989950 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a4e37b_bd42_4ad2_bc86_88e0fcbd7394.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a4e37b_bd42_4ad2_bc86_88e0fcbd7394.slice/crio-ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee\": RecentStats: unable to find data in memory cache]" Sep 11 12:34:23.003670 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.003638 2575 generic.go:356] "Generic (PLEG): container finished" podID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerID="1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600" exitCode=0 Sep 11 12:34:23.003795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.003690 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" Sep 11 12:34:23.003795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.003704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerDied","Data":"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600"} Sep 11 12:34:23.003795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.003782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z" event={"ID":"b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394","Type":"ContainerDied","Data":"ecb3cf978d1016beea08dfbc0de373e3cd18adf3ebc2f29e9354cafc4fcfa4ee"} Sep 11 12:34:23.003911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.003804 2575 scope.go:117] "RemoveContainer" containerID="1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600" Sep 11 12:34:23.013793 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.013762 2575 scope.go:117] "RemoveContainer" containerID="4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521" Sep 11 12:34:23.023538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.023518 2575 scope.go:117] "RemoveContainer" containerID="1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600" Sep 11 12:34:23.023866 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:23.023836 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600\": container with ID starting with 1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600 not found: ID does not exist" containerID="1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600" Sep 11 12:34:23.023978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.023871 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600"} err="failed to get container status \"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600\": rpc error: code = NotFound desc = could not find container \"1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600\": container with ID starting with 1a0f44699546a3bd5feed8f17b122896eccd0343a16043c28c6489a4baeed600 not found: ID does not exist" Sep 11 12:34:23.023978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.023888 2575 scope.go:117] "RemoveContainer" containerID="4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521" Sep 11 12:34:23.024184 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:23.024165 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521\": container with ID starting with 4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521 not found: ID does not exist" containerID="4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521" Sep 11 12:34:23.024233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.024192 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521"} err="failed to get container status \"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521\": rpc error: code = NotFound desc = could not find container \"4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521\": container with ID starting with 4c5939ca5cbbc39c4a0d5e4e27b9a7cbbae32086cccbae4f892be76384aa7521 not found: ID does not exist" Sep 11 12:34:23.025082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.025063 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:34:23.030265 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:23.030244 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-edffad-openshift-default-6b695777dd-25k7z"] Sep 11 12:34:24.953889 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:24.953850 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" path="/var/lib/kubelet/pods/b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394/volumes" Sep 11 12:34:26.083209 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083178 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-baed99"] Sep 11 12:34:26.083592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083579 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.083592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083593 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.083707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083606 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.083707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083626 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.083707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083686 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.083707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.083696 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a4e37b-bd42-4ad2-bc86-88e0fcbd7394" containerName="istio-proxy" Sep 11 12:34:26.087847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.087828 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:26.091814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.091790 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h59p7\"" Sep 11 12:34:26.095309 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.095289 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-baed99"] Sep 11 12:34:26.259947 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.259913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wkn\" (UniqueName: \"kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn\") pod \"test-curl-7600-baed99\" (UID: \"e0aaf3d7-209d-4db7-b991-be70a96bada5\") " pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:26.360999 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.360914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wkn\" (UniqueName: \"kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn\") pod \"test-curl-7600-baed99\" (UID: \"e0aaf3d7-209d-4db7-b991-be70a96bada5\") " pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:26.370159 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.370136 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wkn\" (UniqueName: \"kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn\") pod \"test-curl-7600-baed99\" (UID: \"e0aaf3d7-209d-4db7-b991-be70a96bada5\") " pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:26.399175 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.399149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:26.674026 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.673988 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-baed99"] Sep 11 12:34:26.675894 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:26.675870 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0aaf3d7_209d_4db7_b991_be70a96bada5.slice/crio-b4689dd1e00e7dd8286665022e412ac3ded6ec22efa986289ea5032ddbe4cffb WatchSource:0}: Error finding container b4689dd1e00e7dd8286665022e412ac3ded6ec22efa986289ea5032ddbe4cffb: Status 404 returned error can't find the container with id b4689dd1e00e7dd8286665022e412ac3ded6ec22efa986289ea5032ddbe4cffb Sep 11 12:34:26.678347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:26.678324 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 12:34:27.036508 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:27.036416 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-baed99" event={"ID":"e0aaf3d7-209d-4db7-b991-be70a96bada5","Type":"ContainerStarted","Data":"b4689dd1e00e7dd8286665022e412ac3ded6ec22efa986289ea5032ddbe4cffb"} Sep 11 12:34:28.044699 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.044665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-baed99" event={"ID":"e0aaf3d7-209d-4db7-b991-be70a96bada5","Type":"ContainerStarted","Data":"56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75"} Sep 11 12:34:28.063488 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.063444 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-baed99" podStartSLOduration=1.283158719 podStartE2EDuration="2.063428729s" totalImagesPullingTime="780.27001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:28.061838805 +0000 UTC m=+1207.550043863" watchObservedRunningTime="2026-09-11 12:34:28.063428729 +0000 UTC m=+1207.551633785" Sep 11 12:34:28.716359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.716323 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-baed99"] Sep 11 12:34:28.721592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.721555 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-baed99"] Sep 11 12:34:28.849780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.849708 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7600-ac3ada"] Sep 11 12:34:28.850135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.850124 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aaf3d7-209d-4db7-b991-be70a96bada5" containerName="test-curl-7600-baed99" Sep 11 12:34:28.850183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.850138 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aaf3d7-209d-4db7-b991-be70a96bada5" containerName="test-curl-7600-baed99" Sep 11 12:34:28.850219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.850202 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aaf3d7-209d-4db7-b991-be70a96bada5" containerName="test-curl-7600-baed99" Sep 11 12:34:28.853366 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.853350 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:28.860479 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.860455 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-ac3ada"] Sep 11 12:34:28.986561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:28.986531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7j6p\" (UniqueName: \"kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p\") pod \"test-curl-7600-ac3ada\" (UID: \"1ef60c90-eafd-425f-8fea-a639e61a3a55\") " pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:29.051454 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:29.051430 2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-baed99" container="test-curl-7600-baed99" Sep 11 12:34:29.087428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:29.087397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7j6p\" (UniqueName: \"kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p\") pod \"test-curl-7600-ac3ada\" (UID: \"1ef60c90-eafd-425f-8fea-a639e61a3a55\") " pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:29.096177 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:29.096147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7j6p\" (UniqueName: \"kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p\") pod \"test-curl-7600-ac3ada\" (UID: \"1ef60c90-eafd-425f-8fea-a639e61a3a55\") " pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:29.164299 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:29.164274 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:29.441874 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:29.441845 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7600-ac3ada"] Sep 11 12:34:29.443493 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:29.443465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ef60c90_eafd_425f_8fea_a639e61a3a55.slice/crio-3753e1e451eec339b5e8f6f8df24a67a005599b954a46de9af85a6e4da1fb0e1 WatchSource:0}: Error finding container 3753e1e451eec339b5e8f6f8df24a67a005599b954a46de9af85a6e4da1fb0e1: Status 404 returned error can't find the container with id 3753e1e451eec339b5e8f6f8df24a67a005599b954a46de9af85a6e4da1fb0e1 Sep 11 12:34:30.059900 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.059868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-ac3ada" event={"ID":"1ef60c90-eafd-425f-8fea-a639e61a3a55","Type":"ContainerStarted","Data":"76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921"} Sep 11 12:34:30.060278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.059910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7600-ac3ada" event={"ID":"1ef60c90-eafd-425f-8fea-a639e61a3a55","Type":"ContainerStarted","Data":"3753e1e451eec339b5e8f6f8df24a67a005599b954a46de9af85a6e4da1fb0e1"} Sep 11 12:34:30.060278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.060082 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-baed99" podUID="e0aaf3d7-209d-4db7-b991-be70a96bada5" containerName="test-curl-7600-baed99" containerID="cri-o://56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75" gracePeriod=2 Sep 11 12:34:30.073466 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.073409 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7600-ac3ada" podStartSLOduration=2.073388737 podStartE2EDuration="2.073388737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:30.073024321 +0000 UTC m=+1209.561229379" watchObservedRunningTime="2026-09-11 12:34:30.073388737 +0000 UTC m=+1209.561593794" Sep 11 12:34:30.711071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.711034 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7600-ac3ada"] Sep 11 12:34:30.715037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:30.715003 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7600-ac3ada"] Sep 11 12:34:31.068431 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:31.067992 2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7600-ac3ada" container="test-curl-7600-ac3ada" Sep 11 12:34:31.658410 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.658372 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:34:31.659087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.659063 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef60c90-eafd-425f-8fea-a639e61a3a55" containerName="test-curl-7600-ac3ada" Sep 11 12:34:31.659087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.659088 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef60c90-eafd-425f-8fea-a639e61a3a55" containerName="test-curl-7600-ac3ada" Sep 11 12:34:31.659258 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.659215 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef60c90-eafd-425f-8fea-a639e61a3a55" containerName="test-curl-7600-ac3ada" Sep 11 12:34:31.662685 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.662596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.665280 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.665260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-091983a1-openshift-default-dockercfg-8l2h5\"" Sep 11 12:34:31.674913 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.674890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:34:31.816420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hn5x\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816839 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.816839 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.816815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.917993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hn5x\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918771 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.918864 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.918846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.920829 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.920801 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.921021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.921005 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.927077 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.927056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.927213 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.927192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hn5x\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x\") pod \"e2e-mt-091983a1-openshift-default-7797b8669b-zjg75\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:31.975594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:31.975567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:32.076377 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.076351 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7600-ac3ada" podUID="1ef60c90-eafd-425f-8fea-a639e61a3a55" containerName="test-curl-7600-ac3ada" containerID="cri-o://76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921" gracePeriod=2 Sep 11 12:34:32.225226 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.225203 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:32.253381 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.253355 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:34:32.254582 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:32.254562 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ac0281_f0e7_4f84_8615_cb4cbdc3dece.slice/crio-84d48d344f0608a6a112bf46677cdbbd7f62d0846f4a786e57cf2c4ed924dc0b WatchSource:0}: Error finding container 84d48d344f0608a6a112bf46677cdbbd7f62d0846f4a786e57cf2c4ed924dc0b: Status 404 returned error can't find the container with id 84d48d344f0608a6a112bf46677cdbbd7f62d0846f4a786e57cf2c4ed924dc0b Sep 11 12:34:32.257244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.257215 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:32.257311 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.257279 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:32.257349 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.257307 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:32.322376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.322353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28wkn\" (UniqueName: \"kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn\") pod \"e0aaf3d7-209d-4db7-b991-be70a96bada5\" (UID: \"e0aaf3d7-209d-4db7-b991-be70a96bada5\") " Sep 11 12:34:32.324487 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.324462 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn" (OuterVolumeSpecName: "kube-api-access-28wkn") pod "e0aaf3d7-209d-4db7-b991-be70a96bada5" (UID: "e0aaf3d7-209d-4db7-b991-be70a96bada5"). InnerVolumeSpecName "kube-api-access-28wkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:32.424137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.424093 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28wkn\" (UniqueName: \"kubernetes.io/projected/e0aaf3d7-209d-4db7-b991-be70a96bada5-kube-api-access-28wkn\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:32.954227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:32.954200 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aaf3d7-209d-4db7-b991-be70a96bada5" path="/var/lib/kubelet/pods/e0aaf3d7-209d-4db7-b991-be70a96bada5/volumes" Sep 11 12:34:33.084818 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.084780 2575 generic.go:356] "Generic (PLEG): container finished" podID="e0aaf3d7-209d-4db7-b991-be70a96bada5" containerID="56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75" exitCode=137 Sep 11 12:34:33.085235 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.084819 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-baed99" Sep 11 12:34:33.085235 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.084887 2575 scope.go:117] "RemoveContainer" containerID="56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75" Sep 11 12:34:33.087039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.087005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerStarted","Data":"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e"} Sep 11 12:34:33.087151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.087066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerStarted","Data":"84d48d344f0608a6a112bf46677cdbbd7f62d0846f4a786e57cf2c4ed924dc0b"} Sep 11 12:34:33.095096 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.095076 2575 scope.go:117] "RemoveContainer" containerID="56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75" Sep 11 12:34:33.095357 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:33.095343 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75\": container with ID starting with 56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75 not found: ID does not exist" containerID="56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75" Sep 11 12:34:33.095403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.095365 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75"} err="failed to get container status \"56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75\": rpc error: code = NotFound desc = could not find container \"56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75\": container with ID starting with 56af9ca886d936c473466a58902af9ba95eb28eb2bea5fc7b25fd200704ead75 not found: ID does not exist" Sep 11 12:34:33.111699 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.111655 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podStartSLOduration=2.111643381 podStartE2EDuration="2.111643381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:33.107924581 +0000 UTC m=+1212.596129641" watchObservedRunningTime="2026-09-11 12:34:33.111643381 +0000 UTC m=+1212.599848438" Sep 11 12:34:33.976621 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.976583 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:33.982422 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:33.982398 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:34.095148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.095126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:34.096009 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.095992 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:34:34.259968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.259939 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:34.342054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.342024 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7j6p\" (UniqueName: \"kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p\") pod \"1ef60c90-eafd-425f-8fea-a639e61a3a55\" (UID: \"1ef60c90-eafd-425f-8fea-a639e61a3a55\") " Sep 11 12:34:34.344121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.344093 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p" (OuterVolumeSpecName: "kube-api-access-z7j6p") pod "1ef60c90-eafd-425f-8fea-a639e61a3a55" (UID: "1ef60c90-eafd-425f-8fea-a639e61a3a55"). InnerVolumeSpecName "kube-api-access-z7j6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:34:34.443443 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.443418 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7j6p\" (UniqueName: \"kubernetes.io/projected/1ef60c90-eafd-425f-8fea-a639e61a3a55-kube-api-access-z7j6p\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:34:34.956282 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:34.956248 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef60c90-eafd-425f-8fea-a639e61a3a55" path="/var/lib/kubelet/pods/1ef60c90-eafd-425f-8fea-a639e61a3a55/volumes" Sep 11 12:34:34.995183 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:34.995153 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ef60c90_eafd_425f_8fea_a639e61a3a55.slice\": RecentStats: unable to find data in memory cache]" Sep 11 12:34:35.102539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:35.102507 2575 generic.go:356] "Generic (PLEG): container finished" podID="1ef60c90-eafd-425f-8fea-a639e61a3a55" containerID="76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921" exitCode=137 Sep 11 12:34:35.102986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:35.102569 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7600-ac3ada" Sep 11 12:34:35.102986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:35.102637 2575 scope.go:117] "RemoveContainer" containerID="76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921" Sep 11 12:34:35.112275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:35.112122 2575 scope.go:117] "RemoveContainer" containerID="76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921" Sep 11 12:34:35.112403 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:34:35.112385 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921\": container with ID starting with 76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921 not found: ID does not exist" containerID="76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921" Sep 11 12:34:35.112445 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:35.112409 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921"} err="failed to get container status \"76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921\": rpc error: code = NotFound desc = could not find container \"76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921\": container with ID starting with 76c77634a5f3ab755cc4fe2bc855a9282ea3c03578662dd80472dd9248632921 not found: ID does not exist" Sep 11 12:34:46.423635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.420485 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:34:46.432944 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.432911 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.435251 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.435222 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:34:46.435650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.435596 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f4502d-openshift-default-dockercfg-sfrpx\"" Sep 11 12:34:46.565568 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565533 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565783 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbr9\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.565988 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.565977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666565 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.666794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbr9\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667057 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.666986 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.667239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667524 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.667499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.667903 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.667879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.668120 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.668098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.669734 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.669712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.669898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.669878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.675439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.675383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.675701 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.675683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbr9\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9\") pod \"e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:46.746439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:46.746401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:47.060315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.060286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:34:47.062472 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:34:47.062429 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132e4a13_3d10_4888_8072_e36fe7e55b3e.slice/crio-7e3d8fa271fa3ea8be1b0664ed721ec5accabe100b18f57f1ff2d8dff4f75c0c WatchSource:0}: Error finding container 7e3d8fa271fa3ea8be1b0664ed721ec5accabe100b18f57f1ff2d8dff4f75c0c: Status 404 returned error can't find the container with id 7e3d8fa271fa3ea8be1b0664ed721ec5accabe100b18f57f1ff2d8dff4f75c0c Sep 11 12:34:47.064836 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.064801 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:47.064954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.064872 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:47.064954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.064901 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:34:47.186831 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.186793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerStarted","Data":"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e"} Sep 11 12:34:47.187050 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.186862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerStarted","Data":"7e3d8fa271fa3ea8be1b0664ed721ec5accabe100b18f57f1ff2d8dff4f75c0c"} Sep 11 12:34:47.212535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.212470 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podStartSLOduration=1.21244724 podStartE2EDuration="1.21244724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:34:47.208459419 +0000 UTC m=+1226.696664481" watchObservedRunningTime="2026-09-11 12:34:47.21244724 +0000 UTC m=+1226.700652298" Sep 11 12:34:47.746886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:47.746840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:48.752991 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:48.752958 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:49.204187 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:49.204157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:34:49.205280 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:34:49.205255 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:35:00.286233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.286198 2575 generic.go:356] "Generic (PLEG): container finished" podID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerID="75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e" exitCode=0 Sep 11 12:35:00.286677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.286259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerDied","Data":"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e"} Sep 11 12:35:00.286677 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.286664 2575 scope.go:117] "RemoveContainer" containerID="75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e" Sep 11 12:35:00.287945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.287915 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:00.288042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.287983 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:00.288042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:00.288014 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:01.294960 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:01.294928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerStarted","Data":"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9"} Sep 11 12:35:01.975938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:01.975896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:35:01.977491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:01.977446 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:01.977659 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:01.977521 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:02.977370 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:02.977332 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:02.977858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:02.977398 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:03.977062 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:03.977023 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:03.977269 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:03.977146 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:04.095926 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:04.095871 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:35:04.976847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:04.976809 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:04.977155 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:04.977122 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:05.976107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:05.976066 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:05.976577 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:05.976149 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:06.976582 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:06.976546 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Sep 11 12:35:06.977200 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:06.977167 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Sep 11 12:35:07.987188 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:07.987157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:35:07.994035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:07.994007 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:35:13.385179 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.385149 2575 generic.go:356] "Generic (PLEG): container finished" podID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerID="18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e" exitCode=0 Sep 11 12:35:13.385646 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.385235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerDied","Data":"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e"} Sep 11 12:35:13.385788 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.385770 2575 scope.go:117] "RemoveContainer" containerID="18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e" Sep 11 12:35:13.386966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.386940 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:13.387055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.387009 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:13.387055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.387038 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:35:13.588405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:13.588366 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:35:14.401809 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:14.401768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerStarted","Data":"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50"} Sep 11 12:35:14.747360 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:14.747268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:35:14.748400 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:14.748365 2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 12:35:14.748552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:14.748421 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 12:35:15.747035 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:15.746997 2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 12:35:15.747450 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:15.747057 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 12:35:16.747433 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:16.747398 2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 12:35:16.747941 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:16.747455 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 12:35:17.747529 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:17.747496 2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 12:35:17.748006 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:17.747558 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 12:35:18.746958 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:18.746925 2575 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 12:35:18.747156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:18.746982 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 12:35:19.204225 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:19.204188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:35:19.751371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:19.751338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:35:19.752172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:19.752152 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:35:21.101386 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:21.101360 2575 scope.go:117] "RemoveContainer" containerID="b1245d654e4fac13d80fd93af16fd17ab4c2433fe65eec8dfacc636e9b9f5590" Sep 11 12:35:21.110396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:21.110379 2575 scope.go:117] "RemoveContainer" containerID="a2624fb9a55bdbd92294ca7d8924688f7cd7adf388064fb57a2d074c8399d0d7" Sep 11 12:35:28.610952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.610913 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" podUID="358bb651-bd82-462a-878d-e23ff344e3c0" containerName="main" containerID="cri-o://2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833" gracePeriod=45 Sep 11 12:35:28.866193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.866134 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:35:28.896042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896014 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896055 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896089 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896114 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndnm6\" (UniqueName: \"kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896146 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir\") pod \"358bb651-bd82-462a-878d-e23ff344e3c0\" (UID: \"358bb651-bd82-462a-878d-e23ff344e3c0\") " Sep 11 12:35:28.896396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896349 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home" (OuterVolumeSpecName: "home") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:35:28.896396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896367 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:35:28.896586 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:35:28.896695 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896677 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:28.896753 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896697 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:28.896753 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.896705 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:28.898572 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.898543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:35:28.898572 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.898565 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6" (OuterVolumeSpecName: "kube-api-access-ndnm6") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "kube-api-access-ndnm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:35:28.898888 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.898865 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm" (OuterVolumeSpecName: "dshm") pod "358bb651-bd82-462a-878d-e23ff344e3c0" (UID: "358bb651-bd82-462a-878d-e23ff344e3c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:35:28.997994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.997902 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnm6\" (UniqueName: \"kubernetes.io/projected/358bb651-bd82-462a-878d-e23ff344e3c0-kube-api-access-ndnm6\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:28.997994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.997929 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358bb651-bd82-462a-878d-e23ff344e3c0-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:28.997994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:28.997942 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358bb651-bd82-462a-878d-e23ff344e3c0-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:35:29.505956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.505927 2575 generic.go:356] "Generic (PLEG): container finished" podID="358bb651-bd82-462a-878d-e23ff344e3c0" containerID="2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833" exitCode=0 Sep 11 12:35:29.506140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.505971 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" Sep 11 12:35:29.506140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.506014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" event={"ID":"358bb651-bd82-462a-878d-e23ff344e3c0","Type":"ContainerDied","Data":"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833"} Sep 11 12:35:29.506140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.506062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b" event={"ID":"358bb651-bd82-462a-878d-e23ff344e3c0","Type":"ContainerDied","Data":"7a9285a3867d37cce748b6d5e6c237e8edc2c9af4db719e1f76f4de3319a9de8"} Sep 11 12:35:29.506140 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.506085 2575 scope.go:117] "RemoveContainer" containerID="2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833" Sep 11 12:35:29.516088 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.516069 2575 scope.go:117] "RemoveContainer" containerID="2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833" Sep 11 12:35:29.516357 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:35:29.516342 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833\": container with ID starting with 2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833 not found: ID does not exist" containerID="2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833" Sep 11 12:35:29.516416 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.516364 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833"} err="failed to get container status \"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833\": rpc error: code = NotFound desc = could not find container \"2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833\": container with ID starting with 2ed3de982d42c5f928543b4fdc715a708b7fa2f6c7f26a819e2beb3473268833 not found: ID does not exist" Sep 11 12:35:29.525155 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.525137 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:35:29.527916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:29.527897 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/e2e-conflict-b-97e93ffc-kserve-6d5f7485b7-c576b"] Sep 11 12:35:30.960448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:30.960417 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358bb651-bd82-462a-878d-e23ff344e3c0" path="/var/lib/kubelet/pods/358bb651-bd82-462a-878d-e23ff344e3c0/volumes" Sep 11 12:35:34.987118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.987077 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:35:34.987596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.987497 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358bb651-bd82-462a-878d-e23ff344e3c0" containerName="main" Sep 11 12:35:34.987596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.987509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="358bb651-bd82-462a-878d-e23ff344e3c0" containerName="main" Sep 11 12:35:34.987596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.987579 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="358bb651-bd82-462a-878d-e23ff344e3c0" containerName="main" Sep 11 12:35:34.992165 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.992146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:34.995900 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.995878 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-091983a1\"/\"e2e-lifecycle-model-091983a1-kserve-self-signed-certs\"" Sep 11 12:35:34.996008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.995936 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-091983a1\"/\"kube-root-ca.crt\"" Sep 11 12:35:34.996008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.995957 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-091983a1\"/\"openshift-service-ca.crt\"" Sep 11 12:35:34.996276 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.996256 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-091983a1\"/\"default-dockercfg-vjf5v\"" Sep 11 12:35:34.998931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:34.998906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:35:35.045128 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.045228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.045228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgsm\" (UniqueName: \"kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.045228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.045362 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.045362 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.045315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgsm\" (UniqueName: \"kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146678 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146760 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146928 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146908 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.146986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.146966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.148881 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.148860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.151959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.151937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.154379 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.154356 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgsm\" (UniqueName: \"kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm\") pod \"e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.303079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.302998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:35.595499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:35.595469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:35:35.596545 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:35:35.596521 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8eb67c4c_33e6_40d2_ac2b_cb621cab80f7.slice/crio-e07b3d1670fdeccb4422dd591a81fa4c5998158f97eb019774f2f14b51f42437 WatchSource:0}: Error finding container e07b3d1670fdeccb4422dd591a81fa4c5998158f97eb019774f2f14b51f42437: Status 404 returned error can't find the container with id e07b3d1670fdeccb4422dd591a81fa4c5998158f97eb019774f2f14b51f42437 Sep 11 12:35:36.558933 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:36.558896 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" event={"ID":"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7","Type":"ContainerStarted","Data":"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256"} Sep 11 12:35:36.559316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:36.558945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" event={"ID":"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7","Type":"ContainerStarted","Data":"e07b3d1670fdeccb4422dd591a81fa4c5998158f97eb019774f2f14b51f42437"} Sep 11 12:35:36.578363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:36.578311 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" podStartSLOduration=2.578295874 podStartE2EDuration="2.578295874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:35:36.575156151 +0000 UTC m=+1276.063361209" watchObservedRunningTime="2026-09-11 12:35:36.578295874 +0000 UTC m=+1276.066500931" Sep 11 12:35:45.303134 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:45.303105 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:45.303513 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:45.303168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:45.315367 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:45.315341 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:45.627402 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:45.627331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:35:51.400904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.400873 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:35:51.404434 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.404419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.407246 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.407227 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"kube-root-ca.crt\"" Sep 11 12:35:51.408419 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.408399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"default-dockercfg-679f9\"" Sep 11 12:35:51.408512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.408438 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"openshift-service-ca.crt\"" Sep 11 12:35:51.408512 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.408440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-aaa824\"/\"auth-test-model-aaa824-kserve-self-signed-certs\"" Sep 11 12:35:51.414007 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.413983 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:35:51.498226 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.498355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.498355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.498438 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp26d\" (UniqueName: \"kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.498473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.498473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.498468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599090 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599053 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp26d\" (UniqueName: \"kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599466 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599466 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599585 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.599744 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.599725 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.601554 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.601536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.601858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.601841 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.608330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.608303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp26d\" (UniqueName: \"kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d\") pod \"auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:51.716377 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:51.716313 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:35:52.001812 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:52.001788 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:35:52.004427 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:35:52.004398 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90059818_5eab_43ac_a7e1_5edbdd5f4df1.slice/crio-e4ce3151361c9439c60b7c620410c4d39080570280861eeba68d88f7ea2213c8 WatchSource:0}: Error finding container e4ce3151361c9439c60b7c620410c4d39080570280861eeba68d88f7ea2213c8: Status 404 returned error can't find the container with id e4ce3151361c9439c60b7c620410c4d39080570280861eeba68d88f7ea2213c8 Sep 11 12:35:52.668439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:52.668393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" event={"ID":"90059818-5eab-43ac-a7e1-5edbdd5f4df1","Type":"ContainerStarted","Data":"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c"} Sep 11 12:35:52.668439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:52.668434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" event={"ID":"90059818-5eab-43ac-a7e1-5edbdd5f4df1","Type":"ContainerStarted","Data":"e4ce3151361c9439c60b7c620410c4d39080570280861eeba68d88f7ea2213c8"} Sep 11 12:35:52.688577 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:35:52.688529 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" podStartSLOduration=1.688512442 podStartE2EDuration="1.688512442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:35:52.6856918 +0000 UTC m=+1292.173896863" watchObservedRunningTime="2026-09-11 12:35:52.688512442 +0000 UTC m=+1292.176717545" Sep 11 12:36:01.716632 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:01.716582 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:36:01.717039 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:01.716655 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:36:01.730330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:01.730300 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:36:01.742750 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:01.742717 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:36:03.504658 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.504629 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:36:03.508514 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.508486 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.511658 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.511635 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cb20c8\"/\"distinct-w4-cb20c8-kserve-self-signed-certs\"" Sep 11 12:36:03.519811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.519787 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:36:03.605484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.605692 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605492 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.605692 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.605692 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.605887 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.605887 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.605729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c775\" (UniqueName: \"kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.706530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.706805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c775\" (UniqueName: \"kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.706805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.706805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.706805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.707043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.706854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.707272 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.707242 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.707347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.707311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.707497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.707466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.716479 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.716454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.718526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.718502 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.719388 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.719363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c775\" (UniqueName: \"kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775\") pod \"distinct-w4-cb20c8-kserve-594c56599f-7vpfv\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:03.820763 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:03.820705 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:04.099075 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:04.099053 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:36:04.100544 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:36:04.100522 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d494967_e9e1_467f_9f34_14965ee826fe.slice/crio-9e0d50ae797e92e98566400a2847851616b90440c78aa6c2cf308448bb4dade3 WatchSource:0}: Error finding container 9e0d50ae797e92e98566400a2847851616b90440c78aa6c2cf308448bb4dade3: Status 404 returned error can't find the container with id 9e0d50ae797e92e98566400a2847851616b90440c78aa6c2cf308448bb4dade3 Sep 11 12:36:04.753085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:04.753046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" event={"ID":"5d494967-e9e1-467f-9f34-14965ee826fe","Type":"ContainerStarted","Data":"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4"} Sep 11 12:36:04.753502 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:04.753101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" event={"ID":"5d494967-e9e1-467f-9f34-14965ee826fe","Type":"ContainerStarted","Data":"9e0d50ae797e92e98566400a2847851616b90440c78aa6c2cf308448bb4dade3"} Sep 11 12:36:04.775491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:04.775445 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" podStartSLOduration=1.775430503 podStartE2EDuration="1.775430503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:36:04.773731254 +0000 UTC m=+1304.261936321" watchObservedRunningTime="2026-09-11 12:36:04.775430503 +0000 UTC m=+1304.263635571" Sep 11 12:36:11.199264 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.199219 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn"] Sep 11 12:36:11.203282 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.203258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:11.205895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.205861 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:36:11.206087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.206062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:36:11.211514 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.211482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn"] Sep 11 12:36:11.377363 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.377315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-091983a1-rfltn\" (UID: \"62116c69-d2ac-48d7-a01f-43a4e093c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:11.478144 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.478048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-091983a1-rfltn\" (UID: \"62116c69-d2ac-48d7-a01f-43a4e093c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:11.479081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.479050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-091983a1-rfltn\" (UID: \"62116c69-d2ac-48d7-a01f-43a4e093c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:11.517435 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.517395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:11.799102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.799069 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn"] Sep 11 12:36:11.800034 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:36:11.800008 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62116c69_d2ac_48d7_a01f_43a4e093c020.slice/crio-59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40 WatchSource:0}: Error finding container 59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40: Status 404 returned error can't find the container with id 59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40 Sep 11 12:36:11.812383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:11.812361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" event={"ID":"62116c69-d2ac-48d7-a01f-43a4e093c020","Type":"ContainerStarted","Data":"59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40"} Sep 11 12:36:12.820105 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:12.820020 2575 generic.go:356] "Generic (PLEG): container finished" podID="62116c69-d2ac-48d7-a01f-43a4e093c020" containerID="8a753dbf1eb0a79b01148258dbc36f4ff7fd48238d85f38a6e3c65060a1408f4" exitCode=0 Sep 11 12:36:12.820460 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:12.820107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" event={"ID":"62116c69-d2ac-48d7-a01f-43a4e093c020","Type":"ContainerDied","Data":"8a753dbf1eb0a79b01148258dbc36f4ff7fd48238d85f38a6e3c65060a1408f4"} Sep 11 12:36:13.820952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:13.820905 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:13.821460 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:13.820973 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:13.835126 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:13.835091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:13.994768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:13.994713 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:14.005957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.005921 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca\") pod \"62116c69-d2ac-48d7-a01f-43a4e093c020\" (UID: \"62116c69-d2ac-48d7-a01f-43a4e093c020\") " Sep 11 12:36:14.006336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.006311 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62116c69-d2ac-48d7-a01f-43a4e093c020" (UID: "62116c69-d2ac-48d7-a01f-43a4e093c020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:36:14.107595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.107483 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62116c69-d2ac-48d7-a01f-43a4e093c020-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:14.837427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.837395 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" Sep 11 12:36:14.837936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.837438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn" event={"ID":"62116c69-d2ac-48d7-a01f-43a4e093c020","Type":"ContainerDied","Data":"59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40"} Sep 11 12:36:14.837936 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.837488 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59633adaf527294f459ac1bc1644960502d09e3cd0373a4a4c5d0177ad6b4c40" Sep 11 12:36:14.849041 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:14.849010 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:36:21.150315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:21.150276 2575 scope.go:117] "RemoveContainer" containerID="71a7a1886ebfa39839eafe34ca895cadf9329d3bf1bb966052a135f51235539a" Sep 11 12:36:21.161304 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:21.161279 2575 scope.go:117] "RemoveContainer" containerID="062c4b8dcfc4aa7536c5e895b11fad964422a96d4a475c77473b975760a2ec4a" Sep 11 12:36:21.171522 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:21.171499 2575 scope.go:117] "RemoveContainer" containerID="9fd1c3b7aed6f8444643a1414df6c5cd225081607157ca42e17c75be41da3da5" Sep 11 12:36:21.183746 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:21.183717 2575 scope.go:117] "RemoveContainer" containerID="80b1eab98b5da495d6c7a1f6ccaec48d3d5ba5ee863f27ea7a5293fe376b8b58" Sep 11 12:36:26.956184 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:26.956148 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn"] Sep 11 12:36:26.956797 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:26.956405 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-091983a1-rfltn"] Sep 11 12:36:28.954067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:28.954025 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62116c69-d2ac-48d7-a01f-43a4e093c020" path="/var/lib/kubelet/pods/62116c69-d2ac-48d7-a01f-43a4e093c020/volumes" Sep 11 12:36:35.143260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:35.143220 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:36:50.163769 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.163691 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" podUID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" containerName="main" containerID="cri-o://08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256" gracePeriod=45 Sep 11 12:36:50.421893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.421837 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:36:50.519500 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519513 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519546 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519577 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519676 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519664 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvgsm\" (UniqueName: \"kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519712 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs\") pod \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\" (UID: \"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7\") " Sep 11 12:36:50.519893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519821 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:36:50.519893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519850 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home" (OuterVolumeSpecName: "home") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:36:50.519893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.519880 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:36:50.520160 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.520144 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:50.520213 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.520164 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:50.520213 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.520177 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:50.521761 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.521742 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm" (OuterVolumeSpecName: "dshm") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:36:50.521879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.521859 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm" (OuterVolumeSpecName: "kube-api-access-jvgsm") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "kube-api-access-jvgsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:36:50.521879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.521870 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" (UID: "8eb67c4c-33e6-40d2-ac2b-cb621cab80f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:36:50.621497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.621472 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvgsm\" (UniqueName: \"kubernetes.io/projected/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-kube-api-access-jvgsm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:50.621497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.621494 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:50.621497 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:50.621502 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:36:51.085662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.085632 2575 generic.go:356] "Generic (PLEG): container finished" podID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" containerID="08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256" exitCode=0 Sep 11 12:36:51.085798 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.085680 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" Sep 11 12:36:51.085798 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.085697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" event={"ID":"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7","Type":"ContainerDied","Data":"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256"} Sep 11 12:36:51.085798 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.085739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9" event={"ID":"8eb67c4c-33e6-40d2-ac2b-cb621cab80f7","Type":"ContainerDied","Data":"e07b3d1670fdeccb4422dd591a81fa4c5998158f97eb019774f2f14b51f42437"} Sep 11 12:36:51.085798 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.085755 2575 scope.go:117] "RemoveContainer" containerID="08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256" Sep 11 12:36:51.097964 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.097946 2575 scope.go:117] "RemoveContainer" containerID="08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256" Sep 11 12:36:51.098273 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:36:51.098255 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256\": container with ID starting with 08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256 not found: ID does not exist" containerID="08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256" Sep 11 12:36:51.098327 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.098281 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256"} err="failed to get container status \"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256\": rpc error: code = NotFound desc = could not find container \"08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256\": container with ID starting with 08fff650b6e4e3775ac520eb9d903314b8d073784dc31aa74406c68c215b2256 not found: ID does not exist" Sep 11 12:36:51.112234 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.112213 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:36:51.119154 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:51.119133 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-091983a1/e2e-lifecycle-model-091983a1-kserve-7dcfc7655f-sgsn9"] Sep 11 12:36:52.954846 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:36:52.954803 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" path="/var/lib/kubelet/pods/8eb67c4c-33e6-40d2-ac2b-cb621cab80f7/volumes" Sep 11 12:37:18.228923 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:18.228890 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:37:18.229476 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:18.229121 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" containerID="cri-o://0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9" gracePeriod=30 Sep 11 12:37:21.236627 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:21.236578 2575 scope.go:117] "RemoveContainer" containerID="f01f99d04f08c5f9934cd2b3ea0b02f62ec3f026006f5b4abdf57966d2565d21" Sep 11 12:37:22.992180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:22.992128 2575 patch_prober.go:28] interesting pod/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:37:22.992670 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:22.992197 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:37:23.490150 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.490126 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:37:23.567111 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567033 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567111 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567070 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hn5x\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567121 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567189 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567220 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567308 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567287 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567539 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567325 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.567588 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567514 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:37:23.567707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567676 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:37:23.567752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567707 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:37:23.567909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.567799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.568070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568029 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data" (OuterVolumeSpecName: "istio-data") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:37:23.568070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568059 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket\") pod \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\" (UID: \"61ac0281-f0e7-4f84-8615-cb4cbdc3dece\") " Sep 11 12:37:23.568306 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568285 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:37:23.568509 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568491 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.568545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568518 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.568545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568530 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.568637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568544 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.568637 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.568558 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.569516 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.569496 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:37:23.569672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.569652 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:37:23.569967 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.569945 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token" (OuterVolumeSpecName: "istio-token") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:37:23.570064 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.569969 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x" (OuterVolumeSpecName: "kube-api-access-5hn5x") pod "61ac0281-f0e7-4f84-8615-cb4cbdc3dece" (UID: "61ac0281-f0e7-4f84-8615-cb4cbdc3dece"). InnerVolumeSpecName "kube-api-access-5hn5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:37:23.669446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.669411 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.669446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.669439 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hn5x\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-kube-api-access-5hn5x\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.669446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.669449 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:23.669711 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:23.669458 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ac0281-f0e7-4f84-8615-cb4cbdc3dece-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:37:24.331236 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.331201 2575 generic.go:356] "Generic (PLEG): container finished" podID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerID="0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9" exitCode=0 Sep 11 12:37:24.331649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.331247 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" Sep 11 12:37:24.331649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.331285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerDied","Data":"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9"} Sep 11 12:37:24.331649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.331342 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75" event={"ID":"61ac0281-f0e7-4f84-8615-cb4cbdc3dece","Type":"ContainerDied","Data":"84d48d344f0608a6a112bf46677cdbbd7f62d0846f4a786e57cf2c4ed924dc0b"} Sep 11 12:37:24.331649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.331367 2575 scope.go:117] "RemoveContainer" containerID="0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9" Sep 11 12:37:24.342394 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.342375 2575 scope.go:117] "RemoveContainer" containerID="75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e" Sep 11 12:37:24.351650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.351455 2575 scope.go:117] "RemoveContainer" containerID="0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9" Sep 11 12:37:24.351802 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:37:24.351781 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9\": container with ID starting with 0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9 not found: ID does not exist" containerID="0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9" Sep 11 12:37:24.351869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.351816 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9"} err="failed to get container status \"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9\": rpc error: code = NotFound desc = could not find container \"0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9\": container with ID starting with 0d77570a07ea9b83e3154612f299ed8272c65af1be0a166dc9d383d551351bf9 not found: ID does not exist" Sep 11 12:37:24.351869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.351842 2575 scope.go:117] "RemoveContainer" containerID="75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e" Sep 11 12:37:24.352151 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:37:24.352126 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e\": container with ID starting with 75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e not found: ID does not exist" containerID="75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e" Sep 11 12:37:24.352255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.352156 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e"} err="failed to get container status \"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e\": rpc error: code = NotFound desc = could not find container \"75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e\": container with ID starting with 75aecbed85ad47fc4378c2fa8429d0bb582f461e25c1be738c4f0208b8ab5a8e not found: ID does not exist" Sep 11 12:37:24.360878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.360855 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:37:24.365239 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.365212 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-091983a1-openshift-default-7797b8669b-zjg75"] Sep 11 12:37:24.954664 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:24.954632 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" path="/var/lib/kubelet/pods/61ac0281-f0e7-4f84-8615-cb4cbdc3dece/volumes" Sep 11 12:37:48.505097 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505063 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505503 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" containerName="main" Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505515 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" containerName="main" Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505525 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505530 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505550 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62116c69-d2ac-48d7-a01f-43a4e093c020" containerName="revoke-keys" Sep 11 12:37:48.505598 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505555 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62116c69-d2ac-48d7-a01f-43a4e093c020" containerName="revoke-keys" Sep 11 12:37:48.505821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505644 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="62116c69-d2ac-48d7-a01f-43a4e093c020" containerName="revoke-keys" Sep 11 12:37:48.505821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505653 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:37:48.505821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505661 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:37:48.505821 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.505666 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eb67c4c-33e6-40d2-ac2b-cb621cab80f7" containerName="main" Sep 11 12:37:48.510194 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.510177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.513287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.513267 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a8ee3\"/\"embedding-w0-1a8ee3-kserve-self-signed-certs\"" Sep 11 12:37:48.519629 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.519589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:37:48.525166 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.525268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4x5\" (UniqueName: \"kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.525268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.525447 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.525494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.525536 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.525496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4x5\" (UniqueName: \"kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.626978 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.626915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.627231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.627207 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.627294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.627256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.627427 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.627399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.629327 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.629306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.629480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.629463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.636005 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.635982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4x5\" (UniqueName: \"kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5\") pod \"embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:48.820528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:48.820423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:49.097432 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:49.097396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:37:49.099040 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:37:49.099004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f5ef3c1_7a9c_428c_ab04_b0f652d41073.slice/crio-a3f93cff82f4273ebf95cb2c27e1896993be03acfa1dc8515db1af58934c1595 WatchSource:0}: Error finding container a3f93cff82f4273ebf95cb2c27e1896993be03acfa1dc8515db1af58934c1595: Status 404 returned error can't find the container with id a3f93cff82f4273ebf95cb2c27e1896993be03acfa1dc8515db1af58934c1595 Sep 11 12:37:49.494412 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:49.494373 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" event={"ID":"5f5ef3c1-7a9c-428c-ab04-b0f652d41073","Type":"ContainerStarted","Data":"3a0f1fac4fc18af548f2d680d55ced4c08bd286008750e44221715bdbc715e30"} Sep 11 12:37:49.494412 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:49.494415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" event={"ID":"5f5ef3c1-7a9c-428c-ab04-b0f652d41073","Type":"ContainerStarted","Data":"a3f93cff82f4273ebf95cb2c27e1896993be03acfa1dc8515db1af58934c1595"} Sep 11 12:37:49.520590 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:49.520524 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" podStartSLOduration=1.520503809 podStartE2EDuration="1.520503809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:37:49.520288372 +0000 UTC m=+1409.008493429" watchObservedRunningTime="2026-09-11 12:37:49.520503809 +0000 UTC m=+1409.008708865" Sep 11 12:37:58.821453 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:58.821394 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:58.821994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:58.821472 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:58.834215 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:58.834188 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:37:59.573807 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:37:59.573779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:38:13.311631 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.311501 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:38:13.312577 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.312551 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:38:13.312747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.312604 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ac0281-f0e7-4f84-8615-cb4cbdc3dece" containerName="istio-proxy" Sep 11 12:38:13.316492 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.316468 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.320905 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.320881 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f55824-openshift-default-dockercfg-59fq4\"" Sep 11 12:38:13.334728 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.334690 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:38:13.407409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.407352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.407720 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.407696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.407940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.407907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.407957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.408033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.408085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.408148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.408177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.408356 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.408225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwf48\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.509848 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.509812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.509848 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.509851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.509989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwf48\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510135 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510297 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510297 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510297 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510453 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510453 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510769 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510774 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.510974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.510954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.512526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.512506 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.512822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.512805 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.518409 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.518387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwf48\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.518521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.518461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token\") pod \"e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.635008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.634977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:13.920806 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.920773 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:38:13.922920 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:38:13.922886 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801d5699_1eba_4272_8474_951634be7b2b.slice/crio-fc9bb4ba5d666a8739ce0a844a917b108012c7264c31f9b486094b34c71fec22 WatchSource:0}: Error finding container fc9bb4ba5d666a8739ce0a844a917b108012c7264c31f9b486094b34c71fec22: Status 404 returned error can't find the container with id fc9bb4ba5d666a8739ce0a844a917b108012c7264c31f9b486094b34c71fec22 Sep 11 12:38:13.925550 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.925510 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:13.925672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.925636 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:13.925714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:13.925690 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:14.668116 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:14.668080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerStarted","Data":"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340"} Sep 11 12:38:14.668534 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:14.668135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerStarted","Data":"fc9bb4ba5d666a8739ce0a844a917b108012c7264c31f9b486094b34c71fec22"} Sep 11 12:38:14.701193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:14.701124 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podStartSLOduration=1.701104537 podStartE2EDuration="1.701104537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:14.700513637 +0000 UTC m=+1434.188718697" watchObservedRunningTime="2026-09-11 12:38:14.701104537 +0000 UTC m=+1434.189309593" Sep 11 12:38:15.635364 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:15.635329 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:15.641006 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:15.640971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:15.678476 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:15.678441 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:15.679504 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:15.679477 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:38.848860 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.848829 2575 generic.go:356] "Generic (PLEG): container finished" podID="801d5699-1eba-4272-8474-951634be7b2b" containerID="b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340" exitCode=0 Sep 11 12:38:38.849313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.848887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerDied","Data":"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340"} Sep 11 12:38:38.849437 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.849418 2575 scope.go:117] "RemoveContainer" containerID="b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340" Sep 11 12:38:38.850719 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.850689 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:38.850814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.850755 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:38.850814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:38.850792 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:39.857488 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:39.857449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerStarted","Data":"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603"} Sep 11 12:38:40.636153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:40.636115 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:40.636355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:40.636334 2575 patch_prober.go:28] interesting pod/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 11 12:38:40.636405 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:40.636386 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 11 12:38:41.635554 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:41.635525 2575 patch_prober.go:28] interesting pod/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 11 12:38:41.635970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:41.635594 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 11 12:38:42.359647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.359598 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:38:42.362752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.362732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.365938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.365918 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d18c79-openshift-default-dockercfg-2wp9j\"" Sep 11 12:38:42.385633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.382714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:38:42.525997 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.525963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.526398 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.526664 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.526835 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsktj\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.526921 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526870 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.526986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.527040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.526995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.527097 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.527045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.527097 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.527069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.627973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.627882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.627973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.627939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.627980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628207 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsktj\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628644 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628723 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628836 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628811 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.628966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.628947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.630509 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.630491 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.630747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.630732 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.635473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.635446 2575 patch_prober.go:28] interesting pod/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 11 12:38:42.635687 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.635659 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 11 12:38:42.637011 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.636988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsktj\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.639081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.639023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token\") pod \"e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.683404 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.683086 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:42.988163 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:38:42.988128 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeab5954_4334_4a0e_90cb_d3038d5f053f.slice/crio-1acffb42b4be3312ed8126298dcf9123c261b9f4f20b32bdc86b4fa65e3ea321 WatchSource:0}: Error finding container 1acffb42b4be3312ed8126298dcf9123c261b9f4f20b32bdc86b4fa65e3ea321: Status 404 returned error can't find the container with id 1acffb42b4be3312ed8126298dcf9123c261b9f4f20b32bdc86b4fa65e3ea321 Sep 11 12:38:42.991930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.991420 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:42.991930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.991505 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:42.991930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:42.991545 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:38:43.001873 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.001191 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:38:43.635974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.635939 2575 patch_prober.go:28] interesting pod/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 11 12:38:43.636420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.635991 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 11 12:38:43.897946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.897845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerStarted","Data":"f546fb834baba6d770cefffb4f3452e841cfb5d3c55b8286e2c7c659b8fa7073"} Sep 11 12:38:43.897946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.897916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerStarted","Data":"1acffb42b4be3312ed8126298dcf9123c261b9f4f20b32bdc86b4fa65e3ea321"} Sep 11 12:38:43.929211 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:43.929164 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podStartSLOduration=1.9291508849999999 podStartE2EDuration="1.929150885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:38:43.926552384 +0000 UTC m=+1463.414757439" watchObservedRunningTime="2026-09-11 12:38:43.929150885 +0000 UTC m=+1463.417355940" Sep 11 12:38:44.636261 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.636224 2575 patch_prober.go:28] interesting pod/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Sep 11 12:38:44.636661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.636291 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Sep 11 12:38:44.684474 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.684442 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:44.699184 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.699146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:44.904730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.904645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:44.905571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:44.905548 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:38:45.639281 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:45.639254 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:45.639649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:45.639601 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:45.640245 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:45.640230 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:38:55.982913 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:55.982886 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerID="8a9cfd3b60adb8a8463c4b21778b13a99501ec0cd7382949242831f952f9ff4f" exitCode=137 Sep 11 12:38:55.983238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:55.982956 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerDied","Data":"8a9cfd3b60adb8a8463c4b21778b13a99501ec0cd7382949242831f952f9ff4f"} Sep 11 12:38:55.983278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:55.983257 2575 scope.go:117] "RemoveContainer" containerID="8a9cfd3b60adb8a8463c4b21778b13a99501ec0cd7382949242831f952f9ff4f" Sep 11 12:38:56.991339 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:56.991306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerStarted","Data":"fcc1ef1b960f401a1e71cd6ef17ac7380991abcfeb9687f48dee404eb8d4570d"} Sep 11 12:38:56.991717 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:38:56.991508 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:39:08.001055 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:08.001019 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:39:13.840297 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.840245 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct"] Sep 11 12:39:13.844675 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.844602 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:13.851544 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.851510 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:39:13.851804 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.851777 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:39:13.872850 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.872809 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct"] Sep 11 12:39:13.965921 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:13.965868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f55824-wzxct\" (UID: \"5b02310a-4d28-454f-aac5-3d54c9ec5454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:14.067431 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:14.067337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f55824-wzxct\" (UID: \"5b02310a-4d28-454f-aac5-3d54c9ec5454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:14.068397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:14.068359 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f55824-wzxct\" (UID: \"5b02310a-4d28-454f-aac5-3d54c9ec5454\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:14.156745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:14.156699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:14.454595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:14.454564 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct"] Sep 11 12:39:14.455735 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:39:14.455702 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b02310a_4d28_454f_aac5_3d54c9ec5454.slice/crio-d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af WatchSource:0}: Error finding container d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af: Status 404 returned error can't find the container with id d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af Sep 11 12:39:15.128729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:15.128687 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" event={"ID":"5b02310a-4d28-454f-aac5-3d54c9ec5454","Type":"ContainerStarted","Data":"3479bd8de0e3b5a71c3767a1f4675131853b45691b85ff367c703bf95d5316c6"} Sep 11 12:39:15.129198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:15.128749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" event={"ID":"5b02310a-4d28-454f-aac5-3d54c9ec5454","Type":"ContainerStarted","Data":"d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af"} Sep 11 12:39:15.148338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:15.148276 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" podStartSLOduration=2.148256191 podStartE2EDuration="2.148256191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:39:15.146404667 +0000 UTC m=+1494.634609723" watchObservedRunningTime="2026-09-11 12:39:15.148256191 +0000 UTC m=+1494.636461335" Sep 11 12:39:16.137297 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:16.137256 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b02310a-4d28-454f-aac5-3d54c9ec5454" containerID="3479bd8de0e3b5a71c3767a1f4675131853b45691b85ff367c703bf95d5316c6" exitCode=0 Sep 11 12:39:16.137691 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:16.137353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" event={"ID":"5b02310a-4d28-454f-aac5-3d54c9ec5454","Type":"ContainerDied","Data":"3479bd8de0e3b5a71c3767a1f4675131853b45691b85ff367c703bf95d5316c6"} Sep 11 12:39:17.299354 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:17.299324 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:17.410403 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:17.410299 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca\") pod \"5b02310a-4d28-454f-aac5-3d54c9ec5454\" (UID: \"5b02310a-4d28-454f-aac5-3d54c9ec5454\") " Sep 11 12:39:17.410723 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:17.410699 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b02310a-4d28-454f-aac5-3d54c9ec5454" (UID: "5b02310a-4d28-454f-aac5-3d54c9ec5454"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:39:17.511961 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:17.511926 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b02310a-4d28-454f-aac5-3d54c9ec5454-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:18.153434 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:18.153408 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" Sep 11 12:39:18.153599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:18.153407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct" event={"ID":"5b02310a-4d28-454f-aac5-3d54c9ec5454","Type":"ContainerDied","Data":"d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af"} Sep 11 12:39:18.153599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:18.153537 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6425f29b2c1c6ee1714e1c5fd196f8539fd90d162921d4746d013753c8d94af" Sep 11 12:39:21.046692 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:21.046656 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:39:21.048691 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:21.048667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:39:21.283781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:21.283754 2575 scope.go:117] "RemoveContainer" containerID="111cb515263ce8797ed6ef8260e61b75dcc6c2ee4b0651742706ca1d69b4080e" Sep 11 12:39:21.293791 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:21.293765 2575 scope.go:117] "RemoveContainer" containerID="46faa3174600e736ea216e16c56bfc650e5b1548bc10ca2ed19a8f09ae29fb38" Sep 11 12:39:29.859325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:29.859281 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct"] Sep 11 12:39:29.862180 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:29.862155 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f55824-wzxct"] Sep 11 12:39:30.953558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:30.953528 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b02310a-4d28-454f-aac5-3d54c9ec5454" path="/var/lib/kubelet/pods/5b02310a-4d28-454f-aac5-3d54c9ec5454/volumes" Sep 11 12:39:36.583250 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.583213 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:39:36.583696 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.583484 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" containerID="cri-o://348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603" gracePeriod=30 Sep 11 12:39:36.915909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.915872 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg"] Sep 11 12:39:36.916321 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.916308 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b02310a-4d28-454f-aac5-3d54c9ec5454" containerName="revoke-keys" Sep 11 12:39:36.916366 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.916324 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b02310a-4d28-454f-aac5-3d54c9ec5454" containerName="revoke-keys" Sep 11 12:39:36.916421 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.916412 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b02310a-4d28-454f-aac5-3d54c9ec5454" containerName="revoke-keys" Sep 11 12:39:36.921069 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.921050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:36.923993 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.923967 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:39:36.924136 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.924037 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:39:36.929831 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:36.929643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg"] Sep 11 12:39:37.034952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:37.034915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d18c79-gktbg\" (UID: \"4465c7ec-72c8-4742-acc3-21c8b7ae3292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:37.136344 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:37.136306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d18c79-gktbg\" (UID: \"4465c7ec-72c8-4742-acc3-21c8b7ae3292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:37.136994 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:37.136968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d18c79-gktbg\" (UID: \"4465c7ec-72c8-4742-acc3-21c8b7ae3292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:37.234143 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:37.234068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:37.516204 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:37.516174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg"] Sep 11 12:39:37.517711 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:39:37.517673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4465c7ec_72c8_4742_acc3_21c8b7ae3292.slice/crio-593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2 WatchSource:0}: Error finding container 593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2: Status 404 returned error can't find the container with id 593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2 Sep 11 12:39:38.289465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:38.289432 2575 generic.go:356] "Generic (PLEG): container finished" podID="4465c7ec-72c8-4742-acc3-21c8b7ae3292" containerID="66e0564995a80740d3344c3d807be966a065bfb9e38f9b42ebbd64248ea62536" exitCode=0 Sep 11 12:39:38.289932 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:38.289520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" event={"ID":"4465c7ec-72c8-4742-acc3-21c8b7ae3292","Type":"ContainerDied","Data":"66e0564995a80740d3344c3d807be966a065bfb9e38f9b42ebbd64248ea62536"} Sep 11 12:39:38.289932 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:38.289589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" event={"ID":"4465c7ec-72c8-4742-acc3-21c8b7ae3292","Type":"ContainerStarted","Data":"593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2"} Sep 11 12:39:39.176589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.176548 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:39:39.194210 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.194172 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:39:39.298486 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.298453 2575 generic.go:356] "Generic (PLEG): container finished" podID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerID="f546fb834baba6d770cefffb4f3452e841cfb5d3c55b8286e2c7c659b8fa7073" exitCode=0 Sep 11 12:39:39.298956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.298523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerDied","Data":"f546fb834baba6d770cefffb4f3452e841cfb5d3c55b8286e2c7c659b8fa7073"} Sep 11 12:39:39.299327 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.299053 2575 scope.go:117] "RemoveContainer" containerID="f546fb834baba6d770cefffb4f3452e841cfb5d3c55b8286e2c7c659b8fa7073" Sep 11 12:39:39.300284 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.300256 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:39:39.300397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.300315 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:39:39.300397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.300365 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:39:39.459436 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.459413 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:39.562138 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.562109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca\") pod \"4465c7ec-72c8-4742-acc3-21c8b7ae3292\" (UID: \"4465c7ec-72c8-4742-acc3-21c8b7ae3292\") " Sep 11 12:39:39.562495 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.562471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4465c7ec-72c8-4742-acc3-21c8b7ae3292" (UID: "4465c7ec-72c8-4742-acc3-21c8b7ae3292"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:39:39.663146 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:39.663115 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4465c7ec-72c8-4742-acc3-21c8b7ae3292-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:40.309564 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.309524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerStarted","Data":"862f242f979bbef031390945168a9c65682157e7783096d7310cf030edd7de6d"} Sep 11 12:39:40.317664 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.317593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" event={"ID":"4465c7ec-72c8-4742-acc3-21c8b7ae3292","Type":"ContainerDied","Data":"593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2"} Sep 11 12:39:40.317880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.317681 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593aaafa4ac249476548dff50c4d2e408ef45e4b76cfc6acfba9c2577f0171b2" Sep 11 12:39:40.317880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.317750 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg" Sep 11 12:39:40.684099 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.684052 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:39:40.685463 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.685428 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 12:39:40.685589 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:40.685524 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 12:39:41.684178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.684141 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 12:39:41.684661 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.684218 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 12:39:41.864078 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.864021 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.994966 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995095 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995147 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995198 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995263 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwf48\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995325 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.995636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995377 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token\") pod \"801d5699-1eba-4272-8474-951634be7b2b\" (UID: \"801d5699-1eba-4272-8474-951634be7b2b\") " Sep 11 12:39:41.996233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.995878 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:41.996233 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.996147 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:39:41.996338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.996283 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data" (OuterVolumeSpecName: "istio-data") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:41.997800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.996419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:42.000107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:41.997990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:42.000107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.000055 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token" (OuterVolumeSpecName: "istio-token") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:39:42.001792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.001694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:39:42.002124 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.002088 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48" (OuterVolumeSpecName: "kube-api-access-qwf48") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "kube-api-access-qwf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:39:42.002730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.002706 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "801d5699-1eba-4272-8474-951634be7b2b" (UID: "801d5699-1eba-4272-8474-951634be7b2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097532 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097561 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097573 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801d5699-1eba-4272-8474-951634be7b2b-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097584 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097597 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097641 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097658 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801d5699-1eba-4272-8474-951634be7b2b-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097671 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801d5699-1eba-4272-8474-951634be7b2b-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.097896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.097695 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwf48\" (UniqueName: \"kubernetes.io/projected/801d5699-1eba-4272-8474-951634be7b2b-kube-api-access-qwf48\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:42.332124 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.332087 2575 generic.go:356] "Generic (PLEG): container finished" podID="801d5699-1eba-4272-8474-951634be7b2b" containerID="348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603" exitCode=0 Sep 11 12:39:42.332315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.332138 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" Sep 11 12:39:42.332315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.332167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerDied","Data":"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603"} Sep 11 12:39:42.332315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.332220 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6" event={"ID":"801d5699-1eba-4272-8474-951634be7b2b","Type":"ContainerDied","Data":"fc9bb4ba5d666a8739ce0a844a917b108012c7264c31f9b486094b34c71fec22"} Sep 11 12:39:42.332315 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.332244 2575 scope.go:117] "RemoveContainer" containerID="348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603" Sep 11 12:39:42.343114 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.343096 2575 scope.go:117] "RemoveContainer" containerID="b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340" Sep 11 12:39:42.352239 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.352221 2575 scope.go:117] "RemoveContainer" containerID="348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603" Sep 11 12:39:42.352534 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:39:42.352518 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603\": container with ID starting with 348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603 not found: ID does not exist" containerID="348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603" Sep 11 12:39:42.352584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.352542 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603"} err="failed to get container status \"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603\": rpc error: code = NotFound desc = could not find container \"348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603\": container with ID starting with 348c0059dfe2948fd7562656a4f07b5b5be86941ac914f882f994a575685a603 not found: ID does not exist" Sep 11 12:39:42.352584 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.352558 2575 scope.go:117] "RemoveContainer" containerID="b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340" Sep 11 12:39:42.352821 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:39:42.352802 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340\": container with ID starting with b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340 not found: ID does not exist" containerID="b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340" Sep 11 12:39:42.352884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.352829 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340"} err="failed to get container status \"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340\": rpc error: code = NotFound desc = could not find container \"b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340\": container with ID starting with b2d2ea4e6e19edeba98b489e7d6288f9d8ec1ce5623bf49420247911aa3db340 not found: ID does not exist" Sep 11 12:39:42.355231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.355209 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:39:42.358622 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.358582 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f55824-openshift-default-78cf4c69fb-cq8z6"] Sep 11 12:39:42.683894 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.683847 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 12:39:42.684102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.683916 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 12:39:42.953892 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:42.953794 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801d5699-1eba-4272-8474-951634be7b2b" path="/var/lib/kubelet/pods/801d5699-1eba-4272-8474-951634be7b2b/volumes" Sep 11 12:39:43.684565 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:43.684523 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 12:39:43.684795 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:43.684595 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 12:39:44.684800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:44.684761 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Sep 11 12:39:44.685397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:44.685366 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Sep 11 12:39:44.905383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:44.905341 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:39:45.724580 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:45.724544 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:39:45.726439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:45.726413 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:39:50.144420 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:50.144389 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52"] Sep 11 12:39:50.147166 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:50.147144 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a8ee3-29818830-w7t52"] Sep 11 12:39:50.953397 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:50.953362 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4aceeee-e25f-4234-b51e-57ef53cfb1a7" path="/var/lib/kubelet/pods/d4aceeee-e25f-4234-b51e-57ef53cfb1a7/volumes" Sep 11 12:39:52.833127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:52.833094 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg"] Sep 11 12:39:52.836027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:52.836005 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d18c79-gktbg"] Sep 11 12:39:52.956842 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:52.956814 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4465c7ec-72c8-4742-acc3-21c8b7ae3292" path="/var/lib/kubelet/pods/4465c7ec-72c8-4742-acc3-21c8b7ae3292/volumes" Sep 11 12:39:54.196906 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.196859 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" podUID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" containerName="main" containerID="cri-o://11489f6aa5fb342912fd6d8bcf8d342058ce5970cd83743545034769e0c4922b" gracePeriod=45 Sep 11 12:39:54.217044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.217008 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" podUID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" containerName="main" containerID="cri-o://3a0f1fac4fc18af548f2d680d55ced4c08bd286008750e44221715bdbc715e30" gracePeriod=45 Sep 11 12:39:54.418139 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.418102 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" containerID="3a0f1fac4fc18af548f2d680d55ced4c08bd286008750e44221715bdbc715e30" exitCode=0 Sep 11 12:39:54.418337 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.418186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" event={"ID":"5f5ef3c1-7a9c-428c-ab04-b0f652d41073","Type":"ContainerDied","Data":"3a0f1fac4fc18af548f2d680d55ced4c08bd286008750e44221715bdbc715e30"} Sep 11 12:39:54.420666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.420506 2575 generic.go:356] "Generic (PLEG): container finished" podID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" containerID="11489f6aa5fb342912fd6d8bcf8d342058ce5970cd83743545034769e0c4922b" exitCode=0 Sep 11 12:39:54.420666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.420592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" event={"ID":"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053","Type":"ContainerDied","Data":"11489f6aa5fb342912fd6d8bcf8d342058ce5970cd83743545034769e0c4922b"} Sep 11 12:39:54.502859 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.502831 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:39:54.505662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.505642 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:39:54.528107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528076 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.528287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528118 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.528287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.528287 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.528909 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528253 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.528894 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529102 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529137 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4x5\" (UniqueName: \"kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529166 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home\") pod \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\" (UID: \"5f5ef3c1-7a9c-428c-ab04-b0f652d41073\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529211 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.529383 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529313 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lssd\" (UniqueName: \"kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd\") pod \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\" (UID: \"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053\") " Sep 11 12:39:54.530073 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.529936 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.531195 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.531048 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.531583 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.531331 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home" (OuterVolumeSpecName: "home") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.533075 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.533033 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm" (OuterVolumeSpecName: "dshm") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.535592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.533223 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.535592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.533405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd" (OuterVolumeSpecName: "kube-api-access-2lssd") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "kube-api-access-2lssd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:39:54.535592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.533475 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home" (OuterVolumeSpecName: "home") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.535592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.535049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm" (OuterVolumeSpecName: "dshm") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.535592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.535457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546055 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546080 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546104 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lssd\" (UniqueName: \"kubernetes.io/projected/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-kube-api-access-2lssd\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546120 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546136 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546149 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546166 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546178 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.546190 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.550481 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" (UID: "a0bc8c1d-c77c-4c19-8a6f-fac0525ba053"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:39:54.550725 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.550487 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:39:54.554183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.554097 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5" (OuterVolumeSpecName: "kube-api-access-st4x5") pod "5f5ef3c1-7a9c-428c-ab04-b0f652d41073" (UID: "5f5ef3c1-7a9c-428c-ab04-b0f652d41073"). InnerVolumeSpecName "kube-api-access-st4x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:39:54.647231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.647191 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.647231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.647224 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:54.647231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:54.647234 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4x5\" (UniqueName: \"kubernetes.io/projected/5f5ef3c1-7a9c-428c-ab04-b0f652d41073-kube-api-access-st4x5\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:39:55.428674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.428637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" event={"ID":"5f5ef3c1-7a9c-428c-ab04-b0f652d41073","Type":"ContainerDied","Data":"a3f93cff82f4273ebf95cb2c27e1896993be03acfa1dc8515db1af58934c1595"} Sep 11 12:39:55.428674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.428671 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs" Sep 11 12:39:55.428674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.428692 2575 scope.go:117] "RemoveContainer" containerID="3a0f1fac4fc18af548f2d680d55ced4c08bd286008750e44221715bdbc715e30" Sep 11 12:39:55.430136 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.430110 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" Sep 11 12:39:55.430136 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.430115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg" event={"ID":"a0bc8c1d-c77c-4c19-8a6f-fac0525ba053","Type":"ContainerDied","Data":"38a4a9c638c4b342503475855586798773f49ac2f6ff3054502ed11ecfa38755"} Sep 11 12:39:55.439113 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.439095 2575 scope.go:117] "RemoveContainer" containerID="11489f6aa5fb342912fd6d8bcf8d342058ce5970cd83743545034769e0c4922b" Sep 11 12:39:55.446338 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.446295 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:39:55.450025 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.450004 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/embedding-w0-1a8ee3-kserve-6fc5dd4875-nhjbs"] Sep 11 12:39:55.460249 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.460227 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:39:55.463286 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:55.463266 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a8ee3/distinct-2-w0-1a8ee3-kserve-b6c4698bf-2l6rg"] Sep 11 12:39:56.841739 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:56.841688 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:39:56.842201 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:56.841914 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" containerID="cri-o://103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31" gracePeriod=30 Sep 11 12:39:56.954653 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:56.954591 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" path="/var/lib/kubelet/pods/5f5ef3c1-7a9c-428c-ab04-b0f652d41073/volumes" Sep 11 12:39:56.955511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:56.955483 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" path="/var/lib/kubelet/pods/a0bc8c1d-c77c-4c19-8a6f-fac0525ba053/volumes" Sep 11 12:39:59.271063 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:59.271023 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:39:59.271472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:39:59.271235 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" containerID="cri-o://862f242f979bbef031390945168a9c65682157e7783096d7310cf030edd7de6d" gracePeriod=30 Sep 11 12:40:00.727893 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:00.727852 2575 patch_prober.go:28] interesting pod/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:40:00.728295 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:00.727914 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:40:02.097063 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.097041 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:40:02.227399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227582 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227582 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227582 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227495 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227582 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227529 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svsbp\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227706 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227736 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.227814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227716 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:02.227814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo\") pod \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\" (UID: \"6f1e127a-bab2-405d-bb5c-a0abc677c5b0\") " Sep 11 12:40:02.228079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227776 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:02.228079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227953 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:02.228079 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.227968 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:40:02.228303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228109 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:02.228303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228277 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.228303 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228294 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.228650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228308 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.228650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228323 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.228650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.228334 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.230044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.230012 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:02.230044 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.230021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp" (OuterVolumeSpecName: "kube-api-access-svsbp") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "kube-api-access-svsbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:02.230197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.230069 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:02.230197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.230097 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6f1e127a-bab2-405d-bb5c-a0abc677c5b0" (UID: "6f1e127a-bab2-405d-bb5c-a0abc677c5b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:40:02.329824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.329782 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svsbp\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-kube-api-access-svsbp\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.329824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.329817 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.329824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.329829 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.330067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.329841 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f1e127a-bab2-405d-bb5c-a0abc677c5b0-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:02.481745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.481654 2575 generic.go:356] "Generic (PLEG): container finished" podID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerID="103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31" exitCode=0 Sep 11 12:40:02.481745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.481716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" Sep 11 12:40:02.481745 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.481733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerDied","Data":"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31"} Sep 11 12:40:02.481970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.481788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq" event={"ID":"6f1e127a-bab2-405d-bb5c-a0abc677c5b0","Type":"ContainerDied","Data":"7a97572603cc2d873e325420b34d77c886752da6de04d83892fbb9ab6fa0c599"} Sep 11 12:40:02.481970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.481811 2575 scope.go:117] "RemoveContainer" containerID="103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31" Sep 11 12:40:02.491773 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.491754 2575 scope.go:117] "RemoveContainer" containerID="12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7" Sep 11 12:40:02.500682 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.500663 2575 scope.go:117] "RemoveContainer" containerID="103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31" Sep 11 12:40:02.500958 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:40:02.500940 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31\": container with ID starting with 103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31 not found: ID does not exist" containerID="103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31" Sep 11 12:40:02.501027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.500969 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31"} err="failed to get container status \"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31\": rpc error: code = NotFound desc = could not find container \"103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31\": container with ID starting with 103e4048ba35d1360ac934b7c3b2d3f4a7de4680562f270446417842dbc8fb31 not found: ID does not exist" Sep 11 12:40:02.501027 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.500994 2575 scope.go:117] "RemoveContainer" containerID="12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7" Sep 11 12:40:02.501264 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:40:02.501241 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7\": container with ID starting with 12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7 not found: ID does not exist" containerID="12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7" Sep 11 12:40:02.501341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.501272 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7"} err="failed to get container status \"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7\": rpc error: code = NotFound desc = could not find container \"12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7\": container with ID starting with 12b1337c12f7f425d26dbbaff3ca3d57afde22eb29531103c78d66378e7920b7 not found: ID does not exist" Sep 11 12:40:02.505178 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.505155 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:40:02.511684 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.511664 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a8ee3-openshift-default-746c458fc8-67xtq"] Sep 11 12:40:02.954910 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:02.954878 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" path="/var/lib/kubelet/pods/6f1e127a-bab2-405d-bb5c-a0abc677c5b0/volumes" Sep 11 12:40:04.498384 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.498355 2575 generic.go:356] "Generic (PLEG): container finished" podID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerID="862f242f979bbef031390945168a9c65682157e7783096d7310cf030edd7de6d" exitCode=0 Sep 11 12:40:04.498792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.498428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerDied","Data":"862f242f979bbef031390945168a9c65682157e7783096d7310cf030edd7de6d"} Sep 11 12:40:04.498792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.498488 2575 scope.go:117] "RemoveContainer" containerID="f546fb834baba6d770cefffb4f3452e841cfb5d3c55b8286e2c7c659b8fa7073" Sep 11 12:40:04.534275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.534223 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:40:04.653534 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653727 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653560 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653727 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653640 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsktj\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653727 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653685 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653727 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653717 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653951 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653951 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653788 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653951 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653851 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.653951 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.653896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token\") pod \"eeab5954-4334-4a0e-90cb-d3038d5f053f\" (UID: \"eeab5954-4334-4a0e-90cb-d3038d5f053f\") " Sep 11 12:40:04.654150 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654001 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:04.654150 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654094 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:04.654317 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654291 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:04.654317 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654306 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data" (OuterVolumeSpecName: "istio-data") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:04.654467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:40:04.654633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654556 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.654633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654589 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.654633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654603 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.654633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654635 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeab5954-4334-4a0e-90cb-d3038d5f053f-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.654633 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.654652 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.655919 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.655896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:40:04.656183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.656167 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token" (OuterVolumeSpecName: "istio-token") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:04.656336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.656315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj" (OuterVolumeSpecName: "kube-api-access-rsktj") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "kube-api-access-rsktj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:04.656376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.656316 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eeab5954-4334-4a0e-90cb-d3038d5f053f" (UID: "eeab5954-4334-4a0e-90cb-d3038d5f053f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:04.762737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.756588 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.762737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.756641 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.762737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.756664 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeab5954-4334-4a0e-90cb-d3038d5f053f-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:04.762737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:04.756678 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsktj\" (UniqueName: \"kubernetes.io/projected/eeab5954-4334-4a0e-90cb-d3038d5f053f-kube-api-access-rsktj\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:05.505662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:05.505638 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" Sep 11 12:40:05.506046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:05.505639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw" event={"ID":"eeab5954-4334-4a0e-90cb-d3038d5f053f","Type":"ContainerDied","Data":"1acffb42b4be3312ed8126298dcf9123c261b9f4f20b32bdc86b4fa65e3ea321"} Sep 11 12:40:05.506046 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:05.505762 2575 scope.go:117] "RemoveContainer" containerID="862f242f979bbef031390945168a9c65682157e7783096d7310cf030edd7de6d" Sep 11 12:40:05.531981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:05.531956 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:40:05.537890 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:05.537868 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d18c79-openshift-default-566cf94746-r5gpw"] Sep 11 12:40:06.953956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:06.953921 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" path="/var/lib/kubelet/pods/eeab5954-4334-4a0e-90cb-d3038d5f053f/volumes" Sep 11 12:40:17.574127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:17.574082 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:40:17.574679 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:17.574383 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" containerID="cri-o://1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6" gracePeriod=30 Sep 11 12:40:21.323714 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:21.323684 2575 scope.go:117] "RemoveContainer" containerID="eb1ab6926f4c0992bfbb30674d4816adf1cb33e735ffe9b6eb86cb57a101dbd6" Sep 11 12:40:21.333639 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:21.333586 2575 scope.go:117] "RemoveContainer" containerID="b59e31065386f2b575860025103c59bde11b8ae679947994702d5e8a122ffff1" Sep 11 12:40:21.343662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:21.343640 2575 scope.go:117] "RemoveContainer" containerID="9e5f48b5bf2af4e2ca53ab4c1139c6248746a6b0817aa008435696ca73a817f8" Sep 11 12:40:22.842238 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.842169 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:40:22.938301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938437 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938535 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938529 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938558 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nccd8\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token\") pod \"a767dd78-1d06-4cd8-b86d-575206369ea0\" (UID: \"a767dd78-1d06-4cd8-b86d-575206369ea0\") " Sep 11 12:40:22.938770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938746 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:22.938927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:22.938927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:22.939031 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938963 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:22.939031 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.938974 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:40:22.939152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.939054 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:22.939152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.939073 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:22.939152 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.939086 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:22.941173 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.941141 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8" (OuterVolumeSpecName: "kube-api-access-nccd8") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "kube-api-access-nccd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:22.941534 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.941502 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:40:22.941672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.941588 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:40:22.941672 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:22.941597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a767dd78-1d06-4cd8-b86d-575206369ea0" (UID: "a767dd78-1d06-4cd8-b86d-575206369ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:40:22.974111 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:40:22.974081 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda767dd78_1d06_4cd8_b86d_575206369ea0.slice/crio-b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5\": RecentStats: unable to find data in memory cache]" Sep 11 12:40:23.039857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039818 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.039857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039852 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nccd8\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-kube-api-access-nccd8\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.039857 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039863 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.040119 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039871 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.040119 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039878 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767dd78-1d06-4cd8-b86d-575206369ea0-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.040119 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.039885 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767dd78-1d06-4cd8-b86d-575206369ea0-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:40:23.630331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.630293 2575 generic.go:356] "Generic (PLEG): container finished" podID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerID="1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6" exitCode=0 Sep 11 12:40:23.630526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.630338 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" Sep 11 12:40:23.630526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.630378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerDied","Data":"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6"} Sep 11 12:40:23.630526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.630434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf" event={"ID":"a767dd78-1d06-4cd8-b86d-575206369ea0","Type":"ContainerDied","Data":"b03f7a3a60933e046396481f74b5fd25b3d9dbaff6bb74fa8be3f32b794e9ec5"} Sep 11 12:40:23.630526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.630457 2575 scope.go:117] "RemoveContainer" containerID="1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6" Sep 11 12:40:23.640102 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.640084 2575 scope.go:117] "RemoveContainer" containerID="9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b" Sep 11 12:40:23.649705 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.649675 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:40:23.649924 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.649909 2575 scope.go:117] "RemoveContainer" containerID="1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6" Sep 11 12:40:23.650397 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:40:23.650323 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6\": container with ID starting with 1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6 not found: ID does not exist" containerID="1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6" Sep 11 12:40:23.650546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.650408 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6"} err="failed to get container status \"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6\": rpc error: code = NotFound desc = could not find container \"1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6\": container with ID starting with 1edc7ea9ba742505bcdcc544a23381984a2611c3fea275b101597e4821d4fdc6 not found: ID does not exist" Sep 11 12:40:23.650546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.650457 2575 scope.go:117] "RemoveContainer" containerID="9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b" Sep 11 12:40:23.651001 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:40:23.650973 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b\": container with ID starting with 9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b not found: ID does not exist" containerID="9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b" Sep 11 12:40:23.651098 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.651008 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b"} err="failed to get container status \"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b\": rpc error: code = NotFound desc = could not find container \"9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b\": container with ID starting with 9fbc8f4de59a2bc18fe1d183f29d0ba2f3359c5959c22f8955e320330836e87b not found: ID does not exist" Sep 11 12:40:23.653729 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:23.653704 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-44f372-openshift-default-6854fb855d-rmchf"] Sep 11 12:40:24.953085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:24.953046 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" path="/var/lib/kubelet/pods/a767dd78-1d06-4cd8-b86d-575206369ea0/volumes" Sep 11 12:40:47.126912 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:47.126871 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:40:47.138465 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:47.138437 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:40:58.003571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:58.003538 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7"] Sep 11 12:40:58.006004 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:58.005975 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cb20c8-29818830-hrcp7"] Sep 11 12:40:58.953822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:40:58.953791 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9cb81d6-3eab-40af-9cdf-3d3f2e7814df" path="/var/lib/kubelet/pods/d9cb81d6-3eab-40af-9cdf-3d3f2e7814df/volumes" Sep 11 12:41:02.147028 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.146995 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" podUID="5d494967-e9e1-467f-9f34-14965ee826fe" containerName="main" containerID="cri-o://bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4" gracePeriod=45 Sep 11 12:41:02.160521 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.160495 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" podUID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" containerName="main" containerID="cri-o://68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24" gracePeriod=45 Sep 11 12:41:02.404169 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.404109 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:41:02.432353 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.432332 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:41:02.458259 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458233 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458293 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458411 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458380 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458428 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c775\" (UniqueName: \"kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458456 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458484 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm\") pod \"5d494967-e9e1-467f-9f34-14965ee826fe\" (UID: \"5d494967-e9e1-467f-9f34-14965ee826fe\") " Sep 11 12:41:02.458533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458490 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home" (OuterVolumeSpecName: "home") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.458781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458557 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.458781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458767 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.458911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458893 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.458970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458916 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.458970 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.458930 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.460948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.460920 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:41:02.461142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.461118 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775" (OuterVolumeSpecName: "kube-api-access-9c775") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "kube-api-access-9c775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:41:02.462260 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.462231 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm" (OuterVolumeSpecName: "dshm") pod "5d494967-e9e1-467f-9f34-14965ee826fe" (UID: "5d494967-e9e1-467f-9f34-14965ee826fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.560255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560290 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560309 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560415 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560394 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560453 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjtth\" (UniqueName: \"kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth\") pod \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\" (UID: \"d577715f-55eb-46c1-a837-ed0ddd55d0c5\") " Sep 11 12:41:02.560604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560493 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.560604 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560510 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560681 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home" (OuterVolumeSpecName: "home") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560756 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560769 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560778 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560788 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d494967-e9e1-467f-9f34-14965ee826fe-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560797 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c775\" (UniqueName: \"kubernetes.io/projected/5d494967-e9e1-467f-9f34-14965ee826fe-kube-api-access-9c775\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.560805 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.560804 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d494967-e9e1-467f-9f34-14965ee826fe-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.562407 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.562386 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm" (OuterVolumeSpecName: "dshm") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:02.562485 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.562440 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:41:02.562526 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.562491 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth" (OuterVolumeSpecName: "kube-api-access-wjtth") pod "d577715f-55eb-46c1-a837-ed0ddd55d0c5" (UID: "d577715f-55eb-46c1-a837-ed0ddd55d0c5"). InnerVolumeSpecName "kube-api-access-wjtth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:41:02.661392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.661334 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjtth\" (UniqueName: \"kubernetes.io/projected/d577715f-55eb-46c1-a837-ed0ddd55d0c5-kube-api-access-wjtth\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.661392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.661357 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d577715f-55eb-46c1-a837-ed0ddd55d0c5-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.661392 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.661366 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d577715f-55eb-46c1-a837-ed0ddd55d0c5-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:02.891763 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.891732 2575 generic.go:356] "Generic (PLEG): container finished" podID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" containerID="68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24" exitCode=0 Sep 11 12:41:02.891899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.891771 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" Sep 11 12:41:02.891899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.891812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" event={"ID":"d577715f-55eb-46c1-a837-ed0ddd55d0c5","Type":"ContainerDied","Data":"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24"} Sep 11 12:41:02.891899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.891854 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7" event={"ID":"d577715f-55eb-46c1-a837-ed0ddd55d0c5","Type":"ContainerDied","Data":"615d14393bb093c6af66eadd50f11cc107dcd0719c3e3bce1354097355ccb2d7"} Sep 11 12:41:02.891899 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.891876 2575 scope.go:117] "RemoveContainer" containerID="68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24" Sep 11 12:41:02.893235 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.893214 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d494967-e9e1-467f-9f34-14965ee826fe" containerID="bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4" exitCode=0 Sep 11 12:41:02.893341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.893239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" event={"ID":"5d494967-e9e1-467f-9f34-14965ee826fe","Type":"ContainerDied","Data":"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4"} Sep 11 12:41:02.893341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.893270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" event={"ID":"5d494967-e9e1-467f-9f34-14965ee826fe","Type":"ContainerDied","Data":"9e0d50ae797e92e98566400a2847851616b90440c78aa6c2cf308448bb4dade3"} Sep 11 12:41:02.893341 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.893277 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv" Sep 11 12:41:02.910446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.903134 2575 scope.go:117] "RemoveContainer" containerID="68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24" Sep 11 12:41:02.910446 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:41:02.903435 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24\": container with ID starting with 68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24 not found: ID does not exist" containerID="68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24" Sep 11 12:41:02.910446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.903475 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24"} err="failed to get container status \"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24\": rpc error: code = NotFound desc = could not find container \"68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24\": container with ID starting with 68924f7b007d5188eca0cfecb3d7991f005a47a191be6ec3b6195e4a20c5ed24 not found: ID does not exist" Sep 11 12:41:02.910446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.903500 2575 scope.go:117] "RemoveContainer" containerID="bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4" Sep 11 12:41:02.915707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.915648 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:41:02.916197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.916182 2575 scope.go:117] "RemoveContainer" containerID="bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4" Sep 11 12:41:02.916455 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:41:02.916439 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4\": container with ID starting with bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4 not found: ID does not exist" containerID="bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4" Sep 11 12:41:02.916498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.916462 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4"} err="failed to get container status \"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4\": rpc error: code = NotFound desc = could not find container \"bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4\": container with ID starting with bbac0d38bc31e1fcf271bb96612772c35e5b396e245e7d0594758fbb53c7dff4 not found: ID does not exist" Sep 11 12:41:02.921053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.921031 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/premium-w4-cb20c8-kserve-bc9bf9696-74sd7"] Sep 11 12:41:02.930937 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.930915 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:41:02.933996 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.933977 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cb20c8/distinct-w4-cb20c8-kserve-594c56599f-7vpfv"] Sep 11 12:41:02.953753 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.953729 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d494967-e9e1-467f-9f34-14965ee826fe" path="/var/lib/kubelet/pods/5d494967-e9e1-467f-9f34-14965ee826fe/volumes" Sep 11 12:41:02.954169 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:02.954151 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" path="/var/lib/kubelet/pods/d577715f-55eb-46c1-a837-ed0ddd55d0c5/volumes" Sep 11 12:41:12.749426 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:12.749324 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:41:21.412967 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:21.412937 2575 scope.go:117] "RemoveContainer" containerID="9628798d957c8c47fe3ba196695c3626d34a1c8e45a55fe1a7d416431b06dabb" Sep 11 12:41:27.770886 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:27.770828 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" podUID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" containerName="main" containerID="cri-o://be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c" gracePeriod=45 Sep 11 12:41:28.018301 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.018276 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:41:28.040957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.040882 2575 generic.go:356] "Generic (PLEG): container finished" podID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" containerID="be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c" exitCode=0 Sep 11 12:41:28.040957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.040936 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" Sep 11 12:41:28.041172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.040979 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" event={"ID":"90059818-5eab-43ac-a7e1-5edbdd5f4df1","Type":"ContainerDied","Data":"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c"} Sep 11 12:41:28.041172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.041027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k" event={"ID":"90059818-5eab-43ac-a7e1-5edbdd5f4df1","Type":"ContainerDied","Data":"e4ce3151361c9439c60b7c620410c4d39080570280861eeba68d88f7ea2213c8"} Sep 11 12:41:28.041172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.041049 2575 scope.go:117] "RemoveContainer" containerID="be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c" Sep 11 12:41:28.050143 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.050128 2575 scope.go:117] "RemoveContainer" containerID="be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c" Sep 11 12:41:28.050415 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:41:28.050399 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c\": container with ID starting with be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c not found: ID does not exist" containerID="be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c" Sep 11 12:41:28.050488 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.050421 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c"} err="failed to get container status \"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c\": rpc error: code = NotFound desc = could not find container \"be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c\": container with ID starting with be9f778e280f1057acef6eb9971fcf57023e778dc9669c50f5f3c62ad5c06e4c not found: ID does not exist" Sep 11 12:41:28.083418 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp26d\" (UniqueName: \"kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083452 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083484 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083535 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083580 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083696 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache\") pod \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\" (UID: \"90059818-5eab-43ac-a7e1-5edbdd5f4df1\") " Sep 11 12:41:28.083790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home" (OuterVolumeSpecName: "home") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:28.083877 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083837 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:28.083929 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.083906 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache" (OuterVolumeSpecName: "model-cache") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:28.084060 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.084040 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tmp-dir\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.084107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.084066 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-model-cache\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.084107 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.084079 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-home\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.085775 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.085754 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 12:41:28.085870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.085839 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm" (OuterVolumeSpecName: "dshm") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:41:28.085930 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.085919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d" (OuterVolumeSpecName: "kube-api-access-pp26d") pod "90059818-5eab-43ac-a7e1-5edbdd5f4df1" (UID: "90059818-5eab-43ac-a7e1-5edbdd5f4df1"). InnerVolumeSpecName "kube-api-access-pp26d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:41:28.185254 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.185231 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90059818-5eab-43ac-a7e1-5edbdd5f4df1-tls-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.185254 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.185254 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90059818-5eab-43ac-a7e1-5edbdd5f4df1-dshm\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.185379 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.185267 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp26d\" (UniqueName: \"kubernetes.io/projected/90059818-5eab-43ac-a7e1-5edbdd5f4df1-kube-api-access-pp26d\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:41:28.363897 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.363873 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:41:28.369799 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.369779 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-aaa824/auth-test-model-aaa824-kserve-5cb999c7cf-pwt2k"] Sep 11 12:41:28.953441 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:41:28.953413 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" path="/var/lib/kubelet/pods/90059818-5eab-43ac-a7e1-5edbdd5f4df1/volumes" Sep 11 12:42:21.451449 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:42:21.451420 2575 scope.go:117] "RemoveContainer" containerID="8a753dbf1eb0a79b01148258dbc36f4ff7fd48238d85f38a6e3c65060a1408f4" Sep 11 12:43:31.797149 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797120 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797536 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797548 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797557 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d494967-e9e1-467f-9f34-14965ee826fe" containerName="main" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797562 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d494967-e9e1-467f-9f34-14965ee826fe" containerName="main" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797568 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797573 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797589 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:43:31.797594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797595 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797601 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797622 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797631 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4465c7ec-72c8-4742-acc3-21c8b7ae3292" containerName="revoke-keys" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797636 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4465c7ec-72c8-4742-acc3-21c8b7ae3292" containerName="revoke-keys" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797646 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797650 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797660 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797666 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797672 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797676 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797683 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797693 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797698 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797705 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797710 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797715 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797719 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797774 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797782 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797788 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797793 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797798 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d494967-e9e1-467f-9f34-14965ee826fe" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797802 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5ef3c1-7a9c-428c-ab04-b0f652d41073" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797807 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a767dd78-1d06-4cd8-b86d-575206369ea0" containerName="istio-proxy" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797813 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0bc8c1d-c77c-4c19-8a6f-fac0525ba053" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797818 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="90059818-5eab-43ac-a7e1-5edbdd5f4df1" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797824 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4465c7ec-72c8-4742-acc3-21c8b7ae3292" containerName="revoke-keys" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797828 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d577715f-55eb-46c1-a837-ed0ddd55d0c5" containerName="main" Sep 11 12:43:31.797878 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.797833 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="801d5699-1eba-4272-8474-951634be7b2b" containerName="istio-proxy" Sep 11 12:43:31.800945 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.800926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.804336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.804312 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0a1087-openshift-default-dockercfg-24fjn\"" Sep 11 12:43:31.812855 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.812831 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:43:31.886484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pppvt\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886649 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.886948 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.886715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.987942 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.987905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.987952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.987981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988127 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988342 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988452 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988452 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pppvt\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988556 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.988895 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.988814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.989106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.989022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.989317 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.989294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.990571 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.990553 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.991033 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.991016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.998193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.998170 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:31.998925 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:31.998902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pppvt\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt\") pod \"e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:32.112150 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.112095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:32.390859 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.390834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:43:32.392245 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:43:32.392222 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10db57aa_cbd1_4874_b0d2_719120b98bdb.slice/crio-550d6671ec2cda3cb3adf25f019881aaedb785233bc958e0fa16d76620a4b174 WatchSource:0}: Error finding container 550d6671ec2cda3cb3adf25f019881aaedb785233bc958e0fa16d76620a4b174: Status 404 returned error can't find the container with id 550d6671ec2cda3cb3adf25f019881aaedb785233bc958e0fa16d76620a4b174 Sep 11 12:43:32.394344 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.394312 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:32.394444 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.394379 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:32.394444 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.394409 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:32.774532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.774441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerStarted","Data":"9fbc8cd159198567196a7206b61b4c4882b487d232384b1f817c44d61ba83500"} Sep 11 12:43:32.774532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.774508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerStarted","Data":"550d6671ec2cda3cb3adf25f019881aaedb785233bc958e0fa16d76620a4b174"} Sep 11 12:43:32.798530 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:32.798485 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podStartSLOduration=1.798470659 podStartE2EDuration="1.798470659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:43:32.795420671 +0000 UTC m=+1752.283625751" watchObservedRunningTime="2026-09-11 12:43:32.798470659 +0000 UTC m=+1752.286675714" Sep 11 12:43:33.112662 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:33.112592 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:33.117545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:33.117522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:33.779863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:33.779825 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:33.780747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:33.780725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:48.817371 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.817332 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": read tcp 10.132.0.2:58444->10.132.0.94:15021: read: connection reset by peer" start-of-body= Sep 11 12:43:48.817833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.817412 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": read tcp 10.132.0.2:58444->10.132.0.94:15021: read: connection reset by peer" Sep 11 12:43:48.870538 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.870511 2575 generic.go:356] "Generic (PLEG): container finished" podID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerID="9fbc8cd159198567196a7206b61b4c4882b487d232384b1f817c44d61ba83500" exitCode=0 Sep 11 12:43:48.870670 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.870582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerDied","Data":"9fbc8cd159198567196a7206b61b4c4882b487d232384b1f817c44d61ba83500"} Sep 11 12:43:48.870986 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.870973 2575 scope.go:117] "RemoveContainer" containerID="9fbc8cd159198567196a7206b61b4c4882b487d232384b1f817c44d61ba83500" Sep 11 12:43:48.872167 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.872143 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:48.872242 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.872230 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:48.872278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:48.872265 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:43:49.887291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:49.887249 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerStarted","Data":"95083b236926b83121a3f072846e2ac3792e3f860497165c8f27bae7fbd10b12"} Sep 11 12:43:50.112523 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:50.112485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:50.112763 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:50.112735 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 12:43:50.112847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:50.112809 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 12:43:51.113009 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:51.112977 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 12:43:51.113442 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:51.113033 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 12:43:52.113172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:52.113138 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 12:43:52.113570 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:52.113196 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 12:43:53.113451 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:53.113419 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 12:43:53.113941 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:53.113495 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 12:43:54.112687 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:54.112647 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 12:43:54.112858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:54.112711 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 12:43:55.116147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:55.116119 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:55.116549 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:55.116533 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:43:55.117128 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:43:55.117110 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:44:00.939882 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.939842 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:44:00.940313 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.940299 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:44:00.940361 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.940316 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:44:00.940417 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.940408 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeab5954-4334-4a0e-90cb-d3038d5f053f" containerName="istio-proxy" Sep 11 12:44:00.940460 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.940418 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1e127a-bab2-405d-bb5c-a0abc677c5b0" containerName="istio-proxy" Sep 11 12:44:00.942389 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.942374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:00.945142 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.945117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d5ae2a-openshift-default-dockercfg-wzqlk\"" Sep 11 12:44:00.957815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:00.957794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:44:01.070669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.070847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.070847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvj45\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.070847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.070847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.071018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.071018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.071018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.071018 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.070986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171366 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171343 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvj45\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171580 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.171863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.172030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.172030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.172030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.171993 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.172278 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.172254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.173997 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.173972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.174340 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.174318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.179778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.179747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvj45\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.179938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.179918 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token\") pod \"e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.254656 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.254579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:01.534743 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.534714 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:44:01.537395 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:44:01.537369 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39abeecd_c04d_4dcc_babe_d2e9a0217ac5.slice/crio-3ad033d3ecc9523c77c1dd5a950cec772818a97a84bafc25492bf2660c22b5eb WatchSource:0}: Error finding container 3ad033d3ecc9523c77c1dd5a950cec772818a97a84bafc25492bf2660c22b5eb: Status 404 returned error can't find the container with id 3ad033d3ecc9523c77c1dd5a950cec772818a97a84bafc25492bf2660c22b5eb Sep 11 12:44:01.539673 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.539631 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:01.539758 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.539729 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:01.539796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.539777 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:01.965052 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.965019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerStarted","Data":"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f"} Sep 11 12:44:01.965439 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.965074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerStarted","Data":"3ad033d3ecc9523c77c1dd5a950cec772818a97a84bafc25492bf2660c22b5eb"} Sep 11 12:44:01.987037 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:01.986989 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podStartSLOduration=1.9869737440000002 podStartE2EDuration="1.986973744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:44:01.985374169 +0000 UTC m=+1781.473579229" watchObservedRunningTime="2026-09-11 12:44:01.986973744 +0000 UTC m=+1781.475178798" Sep 11 12:44:02.255275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:02.255207 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:02.259943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:02.259918 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:02.970847 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:02.970822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:02.971593 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:02.971576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:19.012768 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:19.012725 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:44:19.013203 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:19.012814 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:44:21.094461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.094366 2575 generic.go:356] "Generic (PLEG): container finished" podID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerID="248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f" exitCode=0 Sep 11 12:44:21.094461 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.094396 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerDied","Data":"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f"} Sep 11 12:44:21.095006 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.094802 2575 scope.go:117] "RemoveContainer" containerID="248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f" Sep 11 12:44:21.095870 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.095850 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:44:21.096040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.096012 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:21.096137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.096092 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:21.096193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.096138 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:44:21.099817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:21.099797 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:44:22.103911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:22.103879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerStarted","Data":"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd"} Sep 11 12:44:22.255239 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:22.255207 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:22.255414 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:22.255394 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 12:44:22.255458 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:22.255439 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 12:44:23.255875 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:23.255845 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 12:44:23.256239 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:23.255890 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 12:44:24.255730 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:24.255697 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 12:44:24.255931 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:24.255771 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 12:44:25.255480 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:25.255448 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 12:44:25.255669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:25.255516 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 12:44:26.255231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:26.255196 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 12:44:26.255596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:26.255248 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 12:44:27.258528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:27.258499 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:27.258897 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:27.258848 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:27.259486 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:27.259466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:44:34.386326 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:44:34.386296 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:45:00.142330 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.142249 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2"] Sep 11 12:45:00.145995 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.145979 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8"] Sep 11 12:45:00.146117 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.146099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" Sep 11 12:45:00.148482 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.148465 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:45:00.148968 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.148954 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c"] Sep 11 12:45:00.149071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.149059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" Sep 11 12:45:00.152210 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.152191 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" Sep 11 12:45:00.154479 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.154456 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2"] Sep 11 12:45:00.158853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.158835 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8"] Sep 11 12:45:00.159376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.159351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" Sep 11 12:45:00.162058 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.162029 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c"] Sep 11 12:45:00.170034 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.170014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" Sep 11 12:45:00.210323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.210289 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" Sep 11 12:45:00.441463 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.441426 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2"] Sep 11 12:45:00.442925 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:45:00.442900 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916d404_6415_44b6_a0c8_a1c875bd07f4.slice/crio-43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109 WatchSource:0}: Error finding container 43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109: Status 404 returned error can't find the container with id 43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109 Sep 11 12:45:00.462198 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.462175 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8"] Sep 11 12:45:00.463466 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:45:00.463443 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edf7f86_8df9_47c2_88a2_dc6b7f8b3546.slice/crio-a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc WatchSource:0}: Error finding container a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc: Status 404 returned error can't find the container with id a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc Sep 11 12:45:00.508071 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:00.508048 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c"] Sep 11 12:45:00.508646 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:45:00.508620 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e88ba_e59d_4907_8348_43d51a56e72c.slice/crio-510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537 WatchSource:0}: Error finding container 510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537: Status 404 returned error can't find the container with id 510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537 Sep 11 12:45:01.334844 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.334821 2575 generic.go:356] "Generic (PLEG): container finished" podID="5edf7f86-8df9-47c2-88a2-dc6b7f8b3546" containerID="2b463faa3a7d1f973bd4a26de792a92599fedeeaa7d335f2ca4e1fc8611e68d4" exitCode=0 Sep 11 12:45:01.335164 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.334900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" event={"ID":"5edf7f86-8df9-47c2-88a2-dc6b7f8b3546","Type":"ContainerDied","Data":"2b463faa3a7d1f973bd4a26de792a92599fedeeaa7d335f2ca4e1fc8611e68d4"} Sep 11 12:45:01.335164 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.334950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" event={"ID":"5edf7f86-8df9-47c2-88a2-dc6b7f8b3546","Type":"ContainerStarted","Data":"a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc"} Sep 11 12:45:01.336520 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.336501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" event={"ID":"5d5e88ba-e59d-4907-8348-43d51a56e72c","Type":"ContainerStarted","Data":"ca68c1a8f8cc1ed78da411e820cf708d87617b7b917d30e6cee707fd4ee94923"} Sep 11 12:45:01.336654 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.336537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" event={"ID":"5d5e88ba-e59d-4907-8348-43d51a56e72c","Type":"ContainerStarted","Data":"510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537"} Sep 11 12:45:01.337972 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.337936 2575 generic.go:356] "Generic (PLEG): container finished" podID="6916d404-6415-44b6-a0c8-a1c875bd07f4" containerID="08fe923a38f835f50c124e99666ba471224822f356808cdb0ff2de4e68ba7242" exitCode=0 Sep 11 12:45:01.338052 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.337988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" event={"ID":"6916d404-6415-44b6-a0c8-a1c875bd07f4","Type":"ContainerDied","Data":"08fe923a38f835f50c124e99666ba471224822f356808cdb0ff2de4e68ba7242"} Sep 11 12:45:01.338052 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.338025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" event={"ID":"6916d404-6415-44b6-a0c8-a1c875bd07f4","Type":"ContainerStarted","Data":"43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109"} Sep 11 12:45:01.365853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:01.365810 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" podStartSLOduration=1.365798997 podStartE2EDuration="1.365798997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:01.364280751 +0000 UTC m=+1840.852485810" watchObservedRunningTime="2026-09-11 12:45:01.365798997 +0000 UTC m=+1840.854004074" Sep 11 12:45:02.345841 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:02.345814 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d5e88ba-e59d-4907-8348-43d51a56e72c" containerID="ca68c1a8f8cc1ed78da411e820cf708d87617b7b917d30e6cee707fd4ee94923" exitCode=0 Sep 11 12:45:02.346228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:02.345893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" event={"ID":"5d5e88ba-e59d-4907-8348-43d51a56e72c","Type":"ContainerDied","Data":"ca68c1a8f8cc1ed78da411e820cf708d87617b7b917d30e6cee707fd4ee94923"} Sep 11 12:45:02.506802 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:02.506778 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" Sep 11 12:45:02.509865 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:02.509844 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" Sep 11 12:45:03.352592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.352527 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" Sep 11 12:45:03.352592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.352573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8" event={"ID":"5edf7f86-8df9-47c2-88a2-dc6b7f8b3546","Type":"ContainerDied","Data":"a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc"} Sep 11 12:45:03.353009 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.352652 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1da395fa80837aa7cc4c4e2ad7ee45c509735d4d58329c21b18b5206db06fdc" Sep 11 12:45:03.354156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.354078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" event={"ID":"6916d404-6415-44b6-a0c8-a1c875bd07f4","Type":"ContainerDied","Data":"43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109"} Sep 11 12:45:03.354347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.354329 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43807818ce97da636f2352b64ebd6c1a255e64922721f4c3eb2416f714d5b109" Sep 11 12:45:03.354896 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.354110 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2" Sep 11 12:45:03.495693 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:03.495674 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" Sep 11 12:45:04.361794 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:04.361766 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" Sep 11 12:45:04.362137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:04.361785 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c" event={"ID":"5d5e88ba-e59d-4907-8348-43d51a56e72c","Type":"ContainerDied","Data":"510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537"} Sep 11 12:45:04.362137 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:04.361833 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="510f05a608a405040d7c66428f6dc6fbb9f4bbbad6305030db5e03925ea5c537" Sep 11 12:45:21.499903 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:21.499871 2575 scope.go:117] "RemoveContainer" containerID="3479bd8de0e3b5a71c3767a1f4675131853b45691b85ff367c703bf95d5316c6" Sep 11 12:45:25.837484 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:25.837456 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:45:26.512880 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.512840 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:45:26.513722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513687 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5edf7f86-8df9-47c2-88a2-dc6b7f8b3546" containerName="cleanup" Sep 11 12:45:26.513722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513721 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5edf7f86-8df9-47c2-88a2-dc6b7f8b3546" containerName="cleanup" Sep 11 12:45:26.513722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513730 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6916d404-6415-44b6-a0c8-a1c875bd07f4" containerName="cleanup" Sep 11 12:45:26.513722 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513735 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6916d404-6415-44b6-a0c8-a1c875bd07f4" containerName="cleanup" Sep 11 12:45:26.514029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513754 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5e88ba-e59d-4907-8348-43d51a56e72c" containerName="cleanup" Sep 11 12:45:26.514029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513759 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5e88ba-e59d-4907-8348-43d51a56e72c" containerName="cleanup" Sep 11 12:45:26.514029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513841 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5e88ba-e59d-4907-8348-43d51a56e72c" containerName="cleanup" Sep 11 12:45:26.514029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513850 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5edf7f86-8df9-47c2-88a2-dc6b7f8b3546" containerName="cleanup" Sep 11 12:45:26.514029 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.513859 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6916d404-6415-44b6-a0c8-a1c875bd07f4" containerName="cleanup" Sep 11 12:45:26.517066 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.517049 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.519780 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.519758 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-f92540-openshift-default-dockercfg-hqdjx\"" Sep 11 12:45:26.533772 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.533748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:45:26.622396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.622674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.623042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.623042 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.622776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723325 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723552 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723770 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723771 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.723866 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.723836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.724122 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.724096 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.724331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.724302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.726528 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.726507 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.726666 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.726535 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.732510 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.732489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.733065 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.733039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n\") pod \"e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:26.827183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:26.827126 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:27.108183 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.108156 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:45:27.109311 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:45:27.109287 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965faac3_f518_4c97_a5a2_2f15e2ca80c1.slice/crio-f3e4775d40851b5b641a87eb9b913e36d34b0056ef9973278796fcf900e131b3 WatchSource:0}: Error finding container f3e4775d40851b5b641a87eb9b913e36d34b0056ef9973278796fcf900e131b3: Status 404 returned error can't find the container with id f3e4775d40851b5b641a87eb9b913e36d34b0056ef9973278796fcf900e131b3 Sep 11 12:45:27.111606 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.111568 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:27.111707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.111672 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:27.111747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.111713 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:27.500041 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.500014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerStarted","Data":"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538"} Sep 11 12:45:27.500197 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.500062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerStarted","Data":"f3e4775d40851b5b641a87eb9b913e36d34b0056ef9973278796fcf900e131b3"} Sep 11 12:45:27.532966 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.532914 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podStartSLOduration=1.532895632 podStartE2EDuration="1.532895632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:45:27.529104354 +0000 UTC m=+1867.017309421" watchObservedRunningTime="2026-09-11 12:45:27.532895632 +0000 UTC m=+1867.021100688" Sep 11 12:45:27.828234 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:27.828164 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:28.832493 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:28.832469 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:29.513345 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:29.513316 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:29.514294 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:29.514262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:45.554764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:45.554725 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 12:45:45.555155 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:45.554812 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 12:45:47.633536 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.633503 2575 generic.go:356] "Generic (PLEG): container finished" podID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerID="829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538" exitCode=0 Sep 11 12:45:47.633914 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.633574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerDied","Data":"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538"} Sep 11 12:45:47.633956 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.633920 2575 scope.go:117] "RemoveContainer" containerID="829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538" Sep 11 12:45:47.634983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.634960 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:47.635043 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.635025 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:47.635081 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:47.635057 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892132Ki","pods":"250"} Sep 11 12:45:48.641432 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:48.641397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerStarted","Data":"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd"} Sep 11 12:45:48.828008 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:48.827975 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:48.828171 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:48.828152 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 11 12:45:48.828221 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:48.828193 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 11 12:45:49.828219 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:49.828186 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 11 12:45:49.828594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:49.828242 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 11 12:45:50.828448 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:50.828415 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 11 12:45:50.828854 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:50.828480 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 11 12:45:51.828255 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:51.828220 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 11 12:45:51.828473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:51.828281 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 11 12:45:52.828215 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:52.828184 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Sep 11 12:45:52.828457 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:52.828249 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Sep 11 12:45:53.866084 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:53.866046 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:53.866532 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:53.866480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:53.867317 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:53.867297 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:45:55.091322 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.091290 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst"] Sep 11 12:45:55.094733 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.094711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:55.097648 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.097604 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:45:55.097752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.097638 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:45:55.104592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.104573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst"] Sep 11 12:45:55.187733 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.187713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst\" (UID: \"cecbf76b-b5b3-44e1-83b5-ca86e7150b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:55.289085 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.289059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst\" (UID: \"cecbf76b-b5b3-44e1-83b5-ca86e7150b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:55.289636 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.289600 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst\" (UID: \"cecbf76b-b5b3-44e1-83b5-ca86e7150b98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:55.405499 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.405482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:55.681103 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:55.681077 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst"] Sep 11 12:45:55.681683 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:45:55.681652 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecbf76b_b5b3_44e1_83b5_ca86e7150b98.slice/crio-6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539 WatchSource:0}: Error finding container 6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539: Status 404 returned error can't find the container with id 6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539 Sep 11 12:45:56.691743 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:56.691718 2575 generic.go:356] "Generic (PLEG): container finished" podID="cecbf76b-b5b3-44e1-83b5-ca86e7150b98" containerID="18a12b8a5f1d08dbf40d02657ecfd5612b7dd8a8585cee924f4b9f7067d359e8" exitCode=0 Sep 11 12:45:56.692021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:56.691752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" event={"ID":"cecbf76b-b5b3-44e1-83b5-ca86e7150b98","Type":"ContainerDied","Data":"18a12b8a5f1d08dbf40d02657ecfd5612b7dd8a8585cee924f4b9f7067d359e8"} Sep 11 12:45:56.692021 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:56.691799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" event={"ID":"cecbf76b-b5b3-44e1-83b5-ca86e7150b98","Type":"ContainerStarted","Data":"6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539"} Sep 11 12:45:57.826879 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:57.826857 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:57.913883 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:57.913856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca\") pod \"cecbf76b-b5b3-44e1-83b5-ca86e7150b98\" (UID: \"cecbf76b-b5b3-44e1-83b5-ca86e7150b98\") " Sep 11 12:45:57.914172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:57.914152 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cecbf76b-b5b3-44e1-83b5-ca86e7150b98" (UID: "cecbf76b-b5b3-44e1-83b5-ca86e7150b98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:45:58.015292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:58.015243 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cecbf76b-b5b3-44e1-83b5-ca86e7150b98-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:45:58.704777 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:58.704755 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" Sep 11 12:45:58.704911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:58.704754 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst" event={"ID":"cecbf76b-b5b3-44e1-83b5-ca86e7150b98","Type":"ContainerDied","Data":"6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539"} Sep 11 12:45:58.704911 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:45:58.704872 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3b719ca6bc27fdb7501e0bd41ed0908bb5df5d2347aadd8311742b05ad6539" Sep 11 12:46:01.622778 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:01.622746 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst"] Sep 11 12:46:01.625265 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:01.625244 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f92540-nsdst"] Sep 11 12:46:02.952983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:02.952954 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cecbf76b-b5b3-44e1-83b5-ca86e7150b98" path="/var/lib/kubelet/pods/cecbf76b-b5b3-44e1-83b5-ca86e7150b98/volumes" Sep 11 12:46:08.328789 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:08.328752 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:46:08.329147 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:08.329019 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" containerID="cri-o://f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd" gracePeriod=30 Sep 11 12:46:08.868578 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:08.868542 2575 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:46:08.868761 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:08.868632 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:46:11.979148 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.979116 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf"] Sep 11 12:46:11.979769 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.979748 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cecbf76b-b5b3-44e1-83b5-ca86e7150b98" containerName="revoke-keys" Sep 11 12:46:11.979863 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.979771 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cecbf76b-b5b3-44e1-83b5-ca86e7150b98" containerName="revoke-keys" Sep 11 12:46:11.979907 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.979870 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cecbf76b-b5b3-44e1-83b5-ca86e7150b98" containerName="revoke-keys" Sep 11 12:46:11.983755 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.983739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:11.988136 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.988118 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:46:11.988354 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.988341 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:46:11.999251 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:11.999226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf"] Sep 11 12:46:12.133953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.133926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf\" (UID: \"e83591ef-3717-4c34-a08a-73dc0b45e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:12.234869 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.234794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf\" (UID: \"e83591ef-3717-4c34-a08a-73dc0b45e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:12.235316 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.235297 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf\" (UID: \"e83591ef-3717-4c34-a08a-73dc0b45e3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:12.294291 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.294266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:12.589014 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.588988 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf"] Sep 11 12:46:12.590368 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:46:12.590339 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83591ef_3717_4c34_a08a_73dc0b45e3aa.slice/crio-0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821 WatchSource:0}: Error finding container 0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821: Status 404 returned error can't find the container with id 0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821 Sep 11 12:46:12.802674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:12.802637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" event={"ID":"e83591ef-3717-4c34-a08a-73dc0b45e3aa","Type":"ContainerStarted","Data":"0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821"} Sep 11 12:46:13.569146 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.569122 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749521 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749647 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749682 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749704 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749749 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749794 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749820 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.751270 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.749878 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy\") pod \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\" (UID: \"965faac3-f518-4c97-a5a2-2f15e2ca80c1\") " Sep 11 12:46:13.752062 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.751676 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:46:13.752062 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.751952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:46:13.752193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.752042 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:46:13.752193 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.752086 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:46:13.752566 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.752530 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:46:13.753023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.752992 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:46:13.753981 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.753959 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:46:13.754067 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.753975 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n" (OuterVolumeSpecName: "kube-api-access-bww5n") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "kube-api-access-bww5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:46:13.754174 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.754156 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "965faac3-f518-4c97-a5a2-2f15e2ca80c1" (UID: "965faac3-f518-4c97-a5a2-2f15e2ca80c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:46:13.810382 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.810325 2575 generic.go:356] "Generic (PLEG): container finished" podID="e83591ef-3717-4c34-a08a-73dc0b45e3aa" containerID="b474f54c8427027fbe9a7ad37a008c55c0a64132a9f72a315afed823ed700024" exitCode=0 Sep 11 12:46:13.810479 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.810409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" event={"ID":"e83591ef-3717-4c34-a08a-73dc0b45e3aa","Type":"ContainerDied","Data":"b474f54c8427027fbe9a7ad37a008c55c0a64132a9f72a315afed823ed700024"} Sep 11 12:46:13.812036 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.812018 2575 generic.go:356] "Generic (PLEG): container finished" podID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerID="f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd" exitCode=0 Sep 11 12:46:13.812118 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.812057 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" Sep 11 12:46:13.812163 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.812097 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerDied","Data":"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd"} Sep 11 12:46:13.812163 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.812160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm" event={"ID":"965faac3-f518-4c97-a5a2-2f15e2ca80c1","Type":"ContainerDied","Data":"f3e4775d40851b5b641a87eb9b913e36d34b0056ef9973278796fcf900e131b3"} Sep 11 12:46:13.812225 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.812181 2575 scope.go:117] "RemoveContainer" containerID="f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd" Sep 11 12:46:13.830121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.830097 2575 scope.go:117] "RemoveContainer" containerID="829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538" Sep 11 12:46:13.838837 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.838809 2575 scope.go:117] "RemoveContainer" containerID="f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd" Sep 11 12:46:13.839301 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:46:13.839233 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd\": container with ID starting with f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd not found: ID does not exist" containerID="f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd" Sep 11 12:46:13.839635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.839291 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd"} err="failed to get container status \"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd\": rpc error: code = NotFound desc = could not find container \"f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd\": container with ID starting with f7e4c61452b997ce0e7f79d5dbc827efd3e2e8b31296109690d9b6a5c2bdeefd not found: ID does not exist" Sep 11 12:46:13.839635 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.839325 2575 scope.go:117] "RemoveContainer" containerID="829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538" Sep 11 12:46:13.839800 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:46:13.839774 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538\": container with ID starting with 829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538 not found: ID does not exist" containerID="829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538" Sep 11 12:46:13.839843 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.839802 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538"} err="failed to get container status \"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538\": rpc error: code = NotFound desc = could not find container \"829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538\": container with ID starting with 829ba62d1b99b35404776eb023e69839e618e18ec00e738a0693d218c17f8538 not found: ID does not exist" Sep 11 12:46:13.843296 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.842971 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:46:13.846164 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.846144 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-f92540-openshift-default-6566875fb4-thfkm"] Sep 11 12:46:13.851757 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851730 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851764 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851781 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851795 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851812 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851822 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851835 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bww5n\" (UniqueName: \"kubernetes.io/projected/965faac3-f518-4c97-a5a2-2f15e2ca80c1-kube-api-access-bww5n\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851851 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:13.851953 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:13.851863 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965faac3-f518-4c97-a5a2-2f15e2ca80c1-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:14.952996 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:14.952964 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" path="/var/lib/kubelet/pods/965faac3-f518-4c97-a5a2-2f15e2ca80c1/volumes" Sep 11 12:46:14.953541 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:14.953523 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:15.060811 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.060786 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca\") pod \"e83591ef-3717-4c34-a08a-73dc0b45e3aa\" (UID: \"e83591ef-3717-4c34-a08a-73dc0b45e3aa\") " Sep 11 12:46:15.061087 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.061067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e83591ef-3717-4c34-a08a-73dc0b45e3aa" (UID: "e83591ef-3717-4c34-a08a-73dc0b45e3aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:46:15.162314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.162290 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e83591ef-3717-4c34-a08a-73dc0b45e3aa-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:46:15.827494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.827469 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" Sep 11 12:46:15.827494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.827481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf" event={"ID":"e83591ef-3717-4c34-a08a-73dc0b45e3aa","Type":"ContainerDied","Data":"0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821"} Sep 11 12:46:15.827744 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:15.827518 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d99c226209d6a1cbd8379caaf45b2b3397d5a4baa91f90996e4f29d6f879821" Sep 11 12:46:21.522959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:21.522930 2575 scope.go:117] "RemoveContainer" containerID="66e0564995a80740d3344c3d807be966a065bfb9e38f9b42ebbd64248ea62536" Sep 11 12:46:27.500826 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:27.500795 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8"] Sep 11 12:46:27.503815 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:27.503790 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0a1087-29818845-xcrb8"] Sep 11 12:46:27.699104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:27.699070 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf"] Sep 11 12:46:27.702781 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:27.702755 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a1087-dfzjf"] Sep 11 12:46:28.952833 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:28.952803 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5edf7f86-8df9-47c2-88a2-dc6b7f8b3546" path="/var/lib/kubelet/pods/5edf7f86-8df9-47c2-88a2-dc6b7f8b3546/volumes" Sep 11 12:46:28.953184 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:46:28.953083 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e83591ef-3717-4c34-a08a-73dc0b45e3aa" path="/var/lib/kubelet/pods/e83591ef-3717-4c34-a08a-73dc0b45e3aa/volumes" Sep 11 12:47:24.095514 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.095439 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:47:24.095983 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.095723 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" containerID="cri-o://95083b236926b83121a3f072846e2ac3792e3f860497165c8f27bae7fbd10b12" gracePeriod=30 Sep 11 12:47:24.552504 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552473 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l"] Sep 11 12:47:24.552884 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552871 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:47:24.552946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552885 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:47:24.552946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552897 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83591ef-3717-4c34-a08a-73dc0b45e3aa" containerName="revoke-keys" Sep 11 12:47:24.552946 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552902 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83591ef-3717-4c34-a08a-73dc0b45e3aa" containerName="revoke-keys" Sep 11 12:47:24.553040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552969 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:47:24.553040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552979 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:47:24.553040 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.552986 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83591ef-3717-4c34-a08a-73dc0b45e3aa" containerName="revoke-keys" Sep 11 12:47:24.555894 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.555875 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:24.559375 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.559354 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmfnj\"" Sep 11 12:47:24.559683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.559668 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 12:47:24.569454 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.569433 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l"] Sep 11 12:47:24.726473 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.726280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l\" (UID: \"1bb4f344-0f44-4a06-acda-68eef2f60a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:24.827824 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.827721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l\" (UID: \"1bb4f344-0f44-4a06-acda-68eef2f60a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:24.828355 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.828336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l\" (UID: \"1bb4f344-0f44-4a06-acda-68eef2f60a9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:24.865683 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:24.865650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:25.119218 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:25.119144 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 12:47:25.119545 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:25.119212 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 12:47:25.159151 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:25.153129 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l"] Sep 11 12:47:25.159151 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:47:25.155045 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb4f344_0f44_4a06_acda_68eef2f60a9c.slice/crio-1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a WatchSource:0}: Error finding container 1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a: Status 404 returned error can't find the container with id 1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a Sep 11 12:47:25.236015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:25.235975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" event={"ID":"1bb4f344-0f44-4a06-acda-68eef2f60a9c","Type":"ContainerStarted","Data":"1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a"} Sep 11 12:47:26.244605 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:26.244572 2575 generic.go:356] "Generic (PLEG): container finished" podID="1bb4f344-0f44-4a06-acda-68eef2f60a9c" containerID="b4a2aafd58f3247cfb8a4a760a558eafd00e39b026aabe7d199205a91e2e7e66" exitCode=0 Sep 11 12:47:26.244982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:26.244647 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" event={"ID":"1bb4f344-0f44-4a06-acda-68eef2f60a9c","Type":"ContainerDied","Data":"b4a2aafd58f3247cfb8a4a760a558eafd00e39b026aabe7d199205a91e2e7e66"} Sep 11 12:47:27.378491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:27.378466 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:27.550166 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:27.550096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca\") pod \"1bb4f344-0f44-4a06-acda-68eef2f60a9c\" (UID: \"1bb4f344-0f44-4a06-acda-68eef2f60a9c\") " Sep 11 12:47:27.550401 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:27.550379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bb4f344-0f44-4a06-acda-68eef2f60a9c" (UID: "1bb4f344-0f44-4a06-acda-68eef2f60a9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:47:27.651376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:27.651352 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bb4f344-0f44-4a06-acda-68eef2f60a9c-maas-api-service-ca\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:28.263336 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:28.263313 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" Sep 11 12:47:28.263447 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:28.263350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l" event={"ID":"1bb4f344-0f44-4a06-acda-68eef2f60a9c","Type":"ContainerDied","Data":"1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a"} Sep 11 12:47:28.263447 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:28.263419 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f558b8ee6b829e00382e884d244489f45a314fb80dfcee3e97ad264c91c844a" Sep 11 12:47:29.271590 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.271563 2575 generic.go:356] "Generic (PLEG): container finished" podID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerID="95083b236926b83121a3f072846e2ac3792e3f860497165c8f27bae7fbd10b12" exitCode=0 Sep 11 12:47:29.271989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.271653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerDied","Data":"95083b236926b83121a3f072846e2ac3792e3f860497165c8f27bae7fbd10b12"} Sep 11 12:47:29.271989 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.271734 2575 scope.go:117] "RemoveContainer" containerID="9fbc8cd159198567196a7206b61b4c4882b487d232384b1f817c44d61ba83500" Sep 11 12:47:29.352175 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.352154 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:47:29.367507 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367457 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367599 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367582 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367703 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367689 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367800 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367777 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367861 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367828 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pppvt\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367869 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.367916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367878 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:47:29.367916 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.367897 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.368106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368082 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:29.368335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368148 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs\") pod \"10db57aa-cbd1-4874-b0d2-719120b98bdb\" (UID: \"10db57aa-cbd1-4874-b0d2-719120b98bdb\") " Sep 11 12:47:29.368335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368117 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:29.368335 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368195 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:29.368563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:29.368563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368455 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10db57aa-cbd1-4874-b0d2-719120b98bdb-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.368563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368474 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.368563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368487 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.368563 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.368496 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.369831 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.369808 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:29.369927 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.369906 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt" (OuterVolumeSpecName: "kube-api-access-pppvt") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "kube-api-access-pppvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:47:29.370082 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.370067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:47:29.370124 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.370087 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "10db57aa-cbd1-4874-b0d2-719120b98bdb" (UID: "10db57aa-cbd1-4874-b0d2-719120b98bdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:47:29.469749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.469722 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.469749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.469743 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.469749 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.469751 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.469928 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.469758 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10db57aa-cbd1-4874-b0d2-719120b98bdb-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:29.469928 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:29.469767 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pppvt\" (UniqueName: \"kubernetes.io/projected/10db57aa-cbd1-4874-b0d2-719120b98bdb-kube-api-access-pppvt\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:30.278570 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.278545 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" Sep 11 12:47:30.279019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.278587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk" event={"ID":"10db57aa-cbd1-4874-b0d2-719120b98bdb","Type":"ContainerDied","Data":"550d6671ec2cda3cb3adf25f019881aaedb785233bc958e0fa16d76620a4b174"} Sep 11 12:47:30.279019 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.278664 2575 scope.go:117] "RemoveContainer" containerID="95083b236926b83121a3f072846e2ac3792e3f860497165c8f27bae7fbd10b12" Sep 11 12:47:30.316256 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.316226 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:47:30.326737 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.326710 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0a1087-openshift-default-798c866d6d-gvcfk"] Sep 11 12:47:30.954206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:30.954181 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" path="/var/lib/kubelet/pods/10db57aa-cbd1-4874-b0d2-719120b98bdb/volumes" Sep 11 12:47:35.248674 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:35.248642 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c"] Sep 11 12:47:35.252716 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:35.252687 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-d5ae2a-29818845-nxg4c"] Sep 11 12:47:35.724307 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:35.724273 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l"] Sep 11 12:47:35.728104 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:35.728077 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d5ae2a-fgv2l"] Sep 11 12:47:36.953561 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:36.953532 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bb4f344-0f44-4a06-acda-68eef2f60a9c" path="/var/lib/kubelet/pods/1bb4f344-0f44-4a06-acda-68eef2f60a9c/volumes" Sep 11 12:47:36.953952 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:36.953808 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5e88ba-e59d-4907-8348-43d51a56e72c" path="/var/lib/kubelet/pods/5d5e88ba-e59d-4907-8348-43d51a56e72c/volumes" Sep 11 12:47:43.752063 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:43.752029 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:47:43.752573 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:43.752289 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" containerID="cri-o://32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd" gracePeriod=30 Sep 11 12:47:48.993117 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:48.993091 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:47:49.025332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025257 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvj45\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025303 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025367 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025395 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025506 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025781 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:49.025982 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.025793 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:49.026495 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.026383 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:47:49.026495 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.026427 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.026871 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.026783 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:49.026937 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.026880 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data\") pod \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\" (UID: \"39abeecd-c04d-4dcc-babe-d2e9a0217ac5\") " Sep 11 12:47:49.027446 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data" (OuterVolumeSpecName: "istio-data") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:49.027543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027522 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.027603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027542 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.027603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027555 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.027603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027567 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.027603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.027579 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.028206 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.028181 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:47:49.028650 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.028600 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:47:49.029651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.029605 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45" (OuterVolumeSpecName: "kube-api-access-vvj45") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "kube-api-access-vvj45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:47:49.029810 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.029787 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token" (OuterVolumeSpecName: "istio-token") pod "39abeecd-c04d-4dcc-babe-d2e9a0217ac5" (UID: "39abeecd-c04d-4dcc-babe-d2e9a0217ac5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:47:49.128489 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.128466 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvj45\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-kube-api-access-vvj45\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.128489 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.128489 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.128651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.128499 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.128651 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.128508 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39abeecd-c04d-4dcc-babe-d2e9a0217ac5-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:47:49.409244 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.409216 2575 generic.go:356] "Generic (PLEG): container finished" podID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerID="32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd" exitCode=0 Sep 11 12:47:49.409378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.409271 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" Sep 11 12:47:49.409378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.409294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerDied","Data":"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd"} Sep 11 12:47:49.409378 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.409374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf" event={"ID":"39abeecd-c04d-4dcc-babe-d2e9a0217ac5","Type":"ContainerDied","Data":"3ad033d3ecc9523c77c1dd5a950cec772818a97a84bafc25492bf2660c22b5eb"} Sep 11 12:47:49.409500 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.409401 2575 scope.go:117] "RemoveContainer" containerID="32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd" Sep 11 12:47:49.419153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.419138 2575 scope.go:117] "RemoveContainer" containerID="248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f" Sep 11 12:47:49.427543 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.427527 2575 scope.go:117] "RemoveContainer" containerID="32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd" Sep 11 12:47:49.427814 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:47:49.427795 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd\": container with ID starting with 32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd not found: ID does not exist" containerID="32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd" Sep 11 12:47:49.427874 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.427820 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd"} err="failed to get container status \"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd\": rpc error: code = NotFound desc = could not find container \"32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd\": container with ID starting with 32c0d7a79e698e5db2a778169f024d903734bdffa2515c7a08e3418be9d9decd not found: ID does not exist" Sep 11 12:47:49.427874 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.427835 2575 scope.go:117] "RemoveContainer" containerID="248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f" Sep 11 12:47:49.428103 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:47:49.428086 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f\": container with ID starting with 248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f not found: ID does not exist" containerID="248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f" Sep 11 12:47:49.428192 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.428113 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f"} err="failed to get container status \"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f\": rpc error: code = NotFound desc = could not find container \"248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f\": container with ID starting with 248d153edb712710222e8f7e489ded53bc9ab371ade665a7245f10516c26a66f not found: ID does not exist" Sep 11 12:47:49.432156 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.432128 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:47:49.435669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:49.435649 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5ae2a-openshift-default-5ff989d568-wz5kf"] Sep 11 12:47:50.953813 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:47:50.953785 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" path="/var/lib/kubelet/pods/39abeecd-c04d-4dcc-babe-d2e9a0217ac5/volumes" Sep 11 12:48:13.581048 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:13.581015 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2"] Sep 11 12:48:13.586121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:13.586097 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f4502d-29818845-vlbt2"] Sep 11 12:48:14.952569 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:14.952537 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6916d404-6415-44b6-a0c8-a1c875bd07f4" path="/var/lib/kubelet/pods/6916d404-6415-44b6-a0c8-a1c875bd07f4/volumes" Sep 11 12:48:21.006372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:21.006343 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:48:21.006837 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:21.006545 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" containerID="cri-o://add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50" gracePeriod=30 Sep 11 12:48:26.259973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.259949 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:48:26.425756 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425728 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.425904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425768 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.425904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425795 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.425904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425821 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.425904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425848 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcbr9\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.425904 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425876 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.426227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425909 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.426227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425943 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.426227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.425977 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs\") pod \"132e4a13-3d10-4888-8072-e36fe7e55b3e\" (UID: \"132e4a13-3d10-4888-8072-e36fe7e55b3e\") " Sep 11 12:48:26.426227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426025 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:48:26.426227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426046 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:48:26.426472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426245 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:48:26.426472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426267 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-credential-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.426472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426289 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-socket\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.426472 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426408 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:48:26.426644 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.426478 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 12:48:26.428268 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.428239 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9" (OuterVolumeSpecName: "kube-api-access-lcbr9") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "kube-api-access-lcbr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:48:26.428368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.428240 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 12:48:26.428444 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.428429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:48:26.428546 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.428531 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "132e4a13-3d10-4888-8072-e36fe7e55b3e" (UID: "132e4a13-3d10-4888-8072-e36fe7e55b3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:48:26.526940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526915 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-podinfo\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.526940 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526937 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-token\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.527070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526947 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-workload-certs\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.527070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526955 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132e4a13-3d10-4888-8072-e36fe7e55b3e-istiod-ca-cert\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.527070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526962 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-data\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.527070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526971 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbr9\" (UniqueName: \"kubernetes.io/projected/132e4a13-3d10-4888-8072-e36fe7e55b3e-kube-api-access-lcbr9\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.527070 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.526978 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132e4a13-3d10-4888-8072-e36fe7e55b3e-istio-envoy\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:48:26.640752 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.640725 2575 generic.go:356] "Generic (PLEG): container finished" podID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerID="add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50" exitCode=0 Sep 11 12:48:26.640856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.640782 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" Sep 11 12:48:26.640856 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.640814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerDied","Data":"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50"} Sep 11 12:48:26.640973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.640868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq" event={"ID":"132e4a13-3d10-4888-8072-e36fe7e55b3e","Type":"ContainerDied","Data":"7e3d8fa271fa3ea8be1b0664ed721ec5accabe100b18f57f1ff2d8dff4f75c0c"} Sep 11 12:48:26.640973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.640890 2575 scope.go:117] "RemoveContainer" containerID="add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50" Sep 11 12:48:26.650247 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.650231 2575 scope.go:117] "RemoveContainer" containerID="18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e" Sep 11 12:48:26.658399 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.658381 2575 scope.go:117] "RemoveContainer" containerID="add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50" Sep 11 12:48:26.658675 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:48:26.658655 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50\": container with ID starting with add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50 not found: ID does not exist" containerID="add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50" Sep 11 12:48:26.658747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.658680 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50"} err="failed to get container status \"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50\": rpc error: code = NotFound desc = could not find container \"add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50\": container with ID starting with add7b4f796b501f627b328eddcfd831a52868b6943cc4ad842051e2bdf2b1b50 not found: ID does not exist" Sep 11 12:48:26.658747 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.658696 2575 scope.go:117] "RemoveContainer" containerID="18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e" Sep 11 12:48:26.658915 ip-10-0-140-239 kubenswrapper[2575]: E0911 12:48:26.658900 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e\": container with ID starting with 18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e not found: ID does not exist" containerID="18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e" Sep 11 12:48:26.658954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.658920 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e"} err="failed to get container status \"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e\": rpc error: code = NotFound desc = could not find container \"18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e\": container with ID starting with 18b7f3a58a480c341f080e31b6373c41158853b85ac067509001c34bf582a86e not found: ID does not exist" Sep 11 12:48:26.667892 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.667872 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:48:26.670920 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.670900 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f4502d-openshift-default-5d47c67f8d-f7wqq"] Sep 11 12:48:26.952995 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:48:26.952971 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" path="/var/lib/kubelet/pods/132e4a13-3d10-4888-8072-e36fe7e55b3e/volumes" Sep 11 12:49:21.136950 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:49:21.136923 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:49:21.142941 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:49:21.142922 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:50:47.261790 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:47.261758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:50:47.264053 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:47.261937 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" containerID="cri-o://fcc1ef1b960f401a1e71cd6ef17ac7380991abcfeb9687f48dee404eb8d4570d" gracePeriod=10 Sep 11 12:50:48.449547 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.449517 2575 generic.go:356] "Generic (PLEG): container finished" podID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerID="fcc1ef1b960f401a1e71cd6ef17ac7380991abcfeb9687f48dee404eb8d4570d" exitCode=0 Sep 11 12:50:48.449902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.449561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerDied","Data":"fcc1ef1b960f401a1e71cd6ef17ac7380991abcfeb9687f48dee404eb8d4570d"} Sep 11 12:50:48.449902 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.449638 2575 scope.go:117] "RemoveContainer" containerID="8a9cfd3b60adb8a8463c4b21778b13a99501ec0cd7382949242831f952f9ff4f" Sep 11 12:50:48.511917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.511899 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:50:48.609533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.609466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume\") pod \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " Sep 11 12:50:48.609679 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.609550 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq8gr\" (UniqueName: \"kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr\") pod \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\" (UID: \"ef24076a-b7e6-424d-ac7a-6569c254a1bd\") " Sep 11 12:50:48.609843 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.609821 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ef24076a-b7e6-424d-ac7a-6569c254a1bd" (UID: "ef24076a-b7e6-424d-ac7a-6569c254a1bd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 12:50:48.611558 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.611540 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr" (OuterVolumeSpecName: "kube-api-access-mq8gr") pod "ef24076a-b7e6-424d-ac7a-6569c254a1bd" (UID: "ef24076a-b7e6-424d-ac7a-6569c254a1bd"). InnerVolumeSpecName "kube-api-access-mq8gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 12:50:48.711131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.711096 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq8gr\" (UniqueName: \"kubernetes.io/projected/ef24076a-b7e6-424d-ac7a-6569c254a1bd-kube-api-access-mq8gr\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:50:48.711131 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:48.711128 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef24076a-b7e6-424d-ac7a-6569c254a1bd-extensions-socket-volume\") on node \"ip-10-0-140-239.ec2.internal\" DevicePath \"\"" Sep 11 12:50:49.456228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:49.456202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" event={"ID":"ef24076a-b7e6-424d-ac7a-6569c254a1bd","Type":"ContainerDied","Data":"af284534ad609896fa1b4b59ce2322aa52677116200d1e9de0bb337f83feeb86"} Sep 11 12:50:49.456228 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:49.456217 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889" Sep 11 12:50:49.456669 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:49.456259 2575 scope.go:117] "RemoveContainer" containerID="fcc1ef1b960f401a1e71cd6ef17ac7380991abcfeb9687f48dee404eb8d4570d" Sep 11 12:50:49.478680 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:49.478654 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:50:49.483944 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:49.483924 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dd889"] Sep 11 12:50:50.952819 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:50:50.952791 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" path="/var/lib/kubelet/pods/ef24076a-b7e6-424d-ac7a-6569c254a1bd/volumes" Sep 11 12:51:21.599647 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:21.599604 2575 scope.go:117] "RemoveContainer" containerID="2b463faa3a7d1f973bd4a26de792a92599fedeeaa7d335f2ca4e1fc8611e68d4" Sep 11 12:51:21.607900 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:21.607882 2575 scope.go:117] "RemoveContainer" containerID="ca68c1a8f8cc1ed78da411e820cf708d87617b7b917d30e6cee707fd4ee94923" Sep 11 12:51:21.615594 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:21.615578 2575 scope.go:117] "RemoveContainer" containerID="08fe923a38f835f50c124e99666ba471224822f356808cdb0ff2de4e68ba7242" Sep 11 12:51:47.946501 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946429 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c"] Sep 11 12:51:47.946853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946820 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:51:47.946853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946833 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:51:47.946853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946848 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946854 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946862 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946867 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946876 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946884 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="965faac3-f518-4c97-a5a2-2f15e2ca80c1" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946900 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946905 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946911 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946916 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946921 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bb4f344-0f44-4a06-acda-68eef2f60a9c" containerName="revoke-keys" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946925 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bb4f344-0f44-4a06-acda-68eef2f60a9c" containerName="revoke-keys" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946932 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946936 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946941 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.946954 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.946946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947015 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947021 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bb4f344-0f44-4a06-acda-68eef2f60a9c" containerName="revoke-keys" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947027 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947033 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef24076a-b7e6-424d-ac7a-6569c254a1bd" containerName="manager" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947038 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947044 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947048 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:51:47.947376 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.947054 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10db57aa-cbd1-4874-b0d2-719120b98bdb" containerName="istio-proxy" Sep 11 12:51:47.950119 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.950099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:47.952533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.952517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-tt57v\"" Sep 11 12:51:47.962023 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:47.962003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c"] Sep 11 12:51:48.057786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.057762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v7fb\" (UniqueName: \"kubernetes.io/projected/101d54cd-04ee-47e8-bb97-81771b047ae6-kube-api-access-9v7fb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.057890 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.057876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/101d54cd-04ee-47e8-bb97-81771b047ae6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.158851 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.158826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/101d54cd-04ee-47e8-bb97-81771b047ae6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.158938 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.158856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v7fb\" (UniqueName: \"kubernetes.io/projected/101d54cd-04ee-47e8-bb97-81771b047ae6-kube-api-access-9v7fb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.159143 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.159125 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/101d54cd-04ee-47e8-bb97-81771b047ae6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.168030 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.168007 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v7fb\" (UniqueName: \"kubernetes.io/projected/101d54cd-04ee-47e8-bb97-81771b047ae6-kube-api-access-9v7fb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m4b7c\" (UID: \"101d54cd-04ee-47e8-bb97-81771b047ae6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.259332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.259285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.534595 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.534574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c"] Sep 11 12:51:48.537527 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:51:48.537503 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101d54cd_04ee_47e8_bb97_81771b047ae6.slice/crio-b50087c4105d376a15902b27cd735977aa9bbe596d6b3121b869459fa9215931 WatchSource:0}: Error finding container b50087c4105d376a15902b27cd735977aa9bbe596d6b3121b869459fa9215931: Status 404 returned error can't find the container with id b50087c4105d376a15902b27cd735977aa9bbe596d6b3121b869459fa9215931 Sep 11 12:51:48.794511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.794422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" event={"ID":"101d54cd-04ee-47e8-bb97-81771b047ae6","Type":"ContainerStarted","Data":"5e677b4147943ebe9ebce489506f020b9f024e9a8936c9b1ca1041cb130e4fe0"} Sep 11 12:51:48.794511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.794478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" event={"ID":"101d54cd-04ee-47e8-bb97-81771b047ae6","Type":"ContainerStarted","Data":"b50087c4105d376a15902b27cd735977aa9bbe596d6b3121b869459fa9215931"} Sep 11 12:51:48.794511 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.794500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:51:48.817310 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:48.817263 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" podStartSLOduration=1.8172500459999998 podStartE2EDuration="1.817250046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 12:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:51:48.814226681 +0000 UTC m=+2248.302431736" watchObservedRunningTime="2026-09-11 12:51:48.817250046 +0000 UTC m=+2248.305455100" Sep 11 12:51:59.801751 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:51:59.801725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m4b7c" Sep 11 12:52:21.651579 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:52:21.651540 2575 scope.go:117] "RemoveContainer" containerID="b474f54c8427027fbe9a7ad37a008c55c0a64132a9f72a315afed823ed700024" Sep 11 12:52:21.662494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:52:21.662466 2575 scope.go:117] "RemoveContainer" containerID="18a12b8a5f1d08dbf40d02657ecfd5612b7dd8a8585cee924f4b9f7067d359e8" Sep 11 12:54:08.938095 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:08.938068 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ncct6_253e5021-baad-4b06-8d89-6937720fee17/server/0.log" Sep 11 12:54:09.055331 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:09.055308 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-th5cm_24a1f7dd-c7cf-42c2-a91d-7860c102b09b/manager/0.log" Sep 11 12:54:09.407957 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:09.407937 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-z59lw_5bac7e8a-3405-4eea-abc3-2b2f5b3685af/copy-manifests/0.log" Sep 11 12:54:09.420974 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:09.420941 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-z59lw_5bac7e8a-3405-4eea-abc3-2b2f5b3685af/manager/0.log" Sep 11 12:54:09.969575 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:09.969557 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818830-lcsb4_39e45f27-a3f5-4033-83ee-e5cea42eba46/cleanup/0.log" Sep 11 12:54:11.040764 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:11.040742 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sn22m_19aa197f-9e2b-40d0-98b9-eaa0602df333/manager/0.log" Sep 11 12:54:11.373943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:11.373883 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m4b7c_101d54cd-04ee-47e8-bb97-81771b047ae6/manager/0.log" Sep 11 12:54:11.979398 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:11.979375 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7_35ff791b-4607-4634-90e8-5f9d0da4dad6/istio-proxy/0.log" Sep 11 12:54:12.092305 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:12.092285 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5tbwz_e6ec635e-54ba-493d-97ff-c8bf598b4f57/discovery/0.log" Sep 11 12:54:12.317570 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:12.317521 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-lfx7g_708f5282-ef7f-4758-a06e-a1b41a8366aa/kube-auth-proxy/0.log" Sep 11 12:54:12.443817 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:12.443791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/1.log" Sep 11 12:54:12.699343 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:12.699319 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:54:12.812462 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:12.812442 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f4bb8d9b-qnbg7_29c80efb-0d0b-44ba-b8ec-95d061729afa/router/0.log" Sep 11 12:54:13.374707 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:13.374686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-6lh4v_a8761947-ecb4-4dcf-8ae5-de8e5a65c114/main/0.log" Sep 11 12:54:13.591917 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:13.591898 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fqlmg5_d7f8fe7b-1c1b-474e-adbc-442511787ff4/main/0.log" Sep 11 12:54:13.701852 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:13.701835 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-xvdfc_dcdf6376-cb0a-4fd0-96eb-538c4d7103d4/main/0.log" Sep 11 12:54:21.173223 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:21.173200 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:54:21.182596 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:21.182578 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:54:21.697054 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:21.697033 2575 scope.go:117] "RemoveContainer" containerID="b4a2aafd58f3247cfb8a4a760a558eafd00e39b026aabe7d199205a91e2e7e66" Sep 11 12:54:22.008498 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:22.008445 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-glbfl_b2b3ef44-c676-4dbe-bd64-17e7a276afee/global-pull-secret-syncer/0.log" Sep 11 12:54:22.123494 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:22.123473 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbtsc_819dccc0-085c-40a5-b00c-80b876a761db/konnectivity-agent/0.log" Sep 11 12:54:22.235858 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:22.235832 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-239.ec2.internal_66903a53d0aed29fd4572d053e31c420/haproxy/0.log" Sep 11 12:54:27.035385 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:27.035364 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sn22m_19aa197f-9e2b-40d0-98b9-eaa0602df333/manager/0.log" Sep 11 12:54:27.145396 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:27.145372 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m4b7c_101d54cd-04ee-47e8-bb97-81771b047ae6/manager/0.log" Sep 11 12:54:28.928095 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:28.928065 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-lrk7z_d26cd9a3-d26f-4edd-b61e-face2237376e/cluster-monitoring-operator/0.log" Sep 11 12:54:28.954314 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:28.954288 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-db9vj_a54e1906-a3d1-4df9-a8a1-e20d87ae53cb/kube-state-metrics/0.log" Sep 11 12:54:28.973292 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:28.973272 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-db9vj_a54e1906-a3d1-4df9-a8a1-e20d87ae53cb/kube-rbac-proxy-main/0.log" Sep 11 12:54:28.997786 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:28.997765 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-db9vj_a54e1906-a3d1-4df9-a8a1-e20d87ae53cb/kube-rbac-proxy-self/0.log" Sep 11 12:54:29.063375 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.063338 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-wvltp_e848fd29-3e88-4e06-939d-2b3f37588c79/monitoring-plugin/0.log" Sep 11 12:54:29.216822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.216801 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccxfk_7e05a630-d3a8-413c-8fba-ac81f89b3775/node-exporter/0.log" Sep 11 12:54:29.242773 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.242757 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccxfk_7e05a630-d3a8-413c-8fba-ac81f89b3775/kube-rbac-proxy/0.log" Sep 11 12:54:29.278894 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.278872 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ccxfk_7e05a630-d3a8-413c-8fba-ac81f89b3775/init-textfile/0.log" Sep 11 12:54:29.714533 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.714512 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-t549c_7e3e96ec-dd41-41a2-8180-44689fc76046/prometheus-operator/0.log" Sep 11 12:54:29.735428 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:29.735413 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-t549c_7e3e96ec-dd41-41a2-8180-44689fc76046/kube-rbac-proxy/0.log" Sep 11 12:54:31.985323 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:31.985291 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/2.log" Sep 11 12:54:31.989892 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:31.989871 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dbnv2_8a591965-0202-4194-b784-8aa97294024c/console-operator/3.log" Sep 11 12:54:32.307943 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.307891 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2"] Sep 11 12:54:32.308248 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.308238 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:54:32.308289 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.308250 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39abeecd-c04d-4dcc-babe-d2e9a0217ac5" containerName="istio-proxy" Sep 11 12:54:32.308347 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.308339 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="132e4a13-3d10-4888-8072-e36fe7e55b3e" containerName="istio-proxy" Sep 11 12:54:32.311591 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.311574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.314359 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.314342 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zkgjl\"/\"default-dockercfg-dl5jf\"" Sep 11 12:54:32.315368 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.315348 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"openshift-service-ca.crt\"" Sep 11 12:54:32.315467 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.315350 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zkgjl\"/\"kube-root-ca.crt\"" Sep 11 12:54:32.326813 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.326793 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2"] Sep 11 12:54:32.429754 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.429726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-podres\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.429754 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.429760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxfb\" (UniqueName: \"kubernetes.io/projected/f9de8f47-2569-454a-bbc6-a84f0f614d6d-kube-api-access-9bxfb\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.429898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.429784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-sys\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.429898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.429810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-lib-modules\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.429898 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.429836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-proc\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530231 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-podres\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxfb\" (UniqueName: \"kubernetes.io/projected/f9de8f47-2569-454a-bbc6-a84f0f614d6d-kube-api-access-9bxfb\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-sys\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-lib-modules\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530312 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530310 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-proc\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530343 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-sys\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-podres\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530393 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-proc\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.530459 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.530387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f9de8f47-2569-454a-bbc6-a84f0f614d6d-lib-modules\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.540275 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.540257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxfb\" (UniqueName: \"kubernetes.io/projected/f9de8f47-2569-454a-bbc6-a84f0f614d6d-kube-api-access-9bxfb\") pod \"perf-node-gather-daemonset-mpxd2\" (UID: \"f9de8f47-2569-454a-bbc6-a84f0f614d6d\") " pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.621153 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.621099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:32.701121 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.701099 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-rct9g_1a3738d8-352f-430d-8336-91b2319e5f1f/download-server/0.log" Sep 11 12:54:32.892638 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:32.892597 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2"] Sep 11 12:54:32.894164 ip-10-0-140-239 kubenswrapper[2575]: W0911 12:54:32.894139 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9de8f47_2569_454a_bbc6_a84f0f614d6d.slice/crio-99fa55fcef60ea2a978444a65671ad28dc8da86aebd0a31b4721c24de62617da WatchSource:0}: Error finding container 99fa55fcef60ea2a978444a65671ad28dc8da86aebd0a31b4721c24de62617da: Status 404 returned error can't find the container with id 99fa55fcef60ea2a978444a65671ad28dc8da86aebd0a31b4721c24de62617da Sep 11 12:54:33.372548 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:33.372488 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-pnj8v_e8df42ea-beeb-4e7b-92a0-1d23cd64181c/volume-data-source-validator/0.log" Sep 11 12:54:33.690853 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:33.690828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" event={"ID":"f9de8f47-2569-454a-bbc6-a84f0f614d6d","Type":"ContainerStarted","Data":"d2390d436ce090c862fed36f161ecb95fc09816b8be2f4b505f07f3d3641c07e"} Sep 11 12:54:33.690973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:33.690875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" event={"ID":"f9de8f47-2569-454a-bbc6-a84f0f614d6d","Type":"ContainerStarted","Data":"99fa55fcef60ea2a978444a65671ad28dc8da86aebd0a31b4721c24de62617da"} Sep 11 12:54:33.690973 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:33.690920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:33.711773 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:33.711732 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" podStartSLOduration=1.7117185959999999 podStartE2EDuration="1.711718596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 12:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 12:54:33.708780698 +0000 UTC m=+2413.196985752" watchObservedRunningTime="2026-09-11 12:54:33.711718596 +0000 UTC m=+2413.199923660" Sep 11 12:54:34.511041 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:34.511021 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p5kc7_17d41008-bde7-4c58-970c-840b80e7165b/dns/0.log" Sep 11 12:54:34.536523 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:34.536501 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p5kc7_17d41008-bde7-4c58-970c-840b80e7165b/kube-rbac-proxy/0.log" Sep 11 12:54:34.654375 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:34.654352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5wrfm_b34698e3-de0b-4d89-8c31-901d9efea428/dns-node-resolver/0.log" Sep 11 12:54:35.431187 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:35.431162 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w4tw4_0ba65492-6ff4-489f-b229-3fe6896728aa/node-ca/0.log" Sep 11 12:54:36.370814 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.370789 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7_35ff791b-4607-4634-90e8-5f9d0da4dad6/istio-proxy/0.log" Sep 11 12:54:36.620796 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.620776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5tbwz_e6ec635e-54ba-493d-97ff-c8bf598b4f57/discovery/0.log" Sep 11 12:54:36.669592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.669569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-lfx7g_708f5282-ef7f-4758-a06e-a1b41a8366aa/kube-auth-proxy/0.log" Sep 11 12:54:36.697603 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.697585 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/0.log" Sep 11 12:54:36.700332 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.700316 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/1.log" Sep 11 12:54:36.772230 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.772208 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:54:36.796013 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:36.795998 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f4bb8d9b-qnbg7_29c80efb-0d0b-44ba-b8ec-95d061729afa/router/0.log" Sep 11 12:54:37.747959 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:37.747936 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r9tvs_ecdce906-182f-43e2-b55a-6269548d60b6/serve-healthcheck-canary/0.log" Sep 11 12:54:38.288831 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:38.288807 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s9tjz_0874441a-169d-48eb-9424-330a62809713/insights-operator/0.log" Sep 11 12:54:38.289398 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:38.289351 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s9tjz_0874441a-169d-48eb-9424-330a62809713/insights-operator/1.log" Sep 11 12:54:38.404735 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:38.404712 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kxnc_e58f653d-7336-4495-bff1-a098156d102c/kube-rbac-proxy/0.log" Sep 11 12:54:38.425374 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:38.425351 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kxnc_e58f653d-7336-4495-bff1-a098156d102c/exporter/0.log" Sep 11 12:54:38.446822 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:38.446804 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4kxnc_e58f653d-7336-4495-bff1-a098156d102c/extractor/0.log" Sep 11 12:54:39.705372 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:39.705349 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zkgjl/perf-node-gather-daemonset-mpxd2" Sep 11 12:54:41.325696 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:41.325677 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ncct6_253e5021-baad-4b06-8d89-6937720fee17/server/0.log" Sep 11 12:54:41.411792 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:41.411774 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-th5cm_24a1f7dd-c7cf-42c2-a91d-7860c102b09b/manager/0.log" Sep 11 12:54:41.628491 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:41.628470 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-z59lw_5bac7e8a-3405-4eea-abc3-2b2f5b3685af/manager/0.log" Sep 11 12:54:41.647172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:41.647154 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8dd64745b-z59lw_5bac7e8a-3405-4eea-abc3-2b2f5b3685af/copy-manifests/0.log" Sep 11 12:54:42.563015 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.562988 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7_35ff791b-4607-4634-90e8-5f9d0da4dad6/istio-proxy/0.log" Sep 11 12:54:42.827702 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.827633 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5tbwz_e6ec635e-54ba-493d-97ff-c8bf598b4f57/discovery/0.log" Sep 11 12:54:42.856476 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.856459 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-lfx7g_708f5282-ef7f-4758-a06e-a1b41a8366aa/kube-auth-proxy/0.log" Sep 11 12:54:42.877825 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.877762 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/0.log" Sep 11 12:54:42.880592 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.880575 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/1.log" Sep 11 12:54:42.931227 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.931210 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:54:42.945977 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:42.945959 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f4bb8d9b-qnbg7_29c80efb-0d0b-44ba-b8ec-95d061729afa/router/0.log" Sep 11 12:54:44.474874 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.474851 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hn5b7_35ff791b-4607-4634-90e8-5f9d0da4dad6/istio-proxy/0.log" Sep 11 12:54:44.738106 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.738025 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-5tbwz_e6ec635e-54ba-493d-97ff-c8bf598b4f57/discovery/0.log" Sep 11 12:54:44.784890 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.784868 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b5f595c8f-lfx7g_708f5282-ef7f-4758-a06e-a1b41a8366aa/kube-auth-proxy/0.log" Sep 11 12:54:44.818216 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.818190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/0.log" Sep 11 12:54:44.820172 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.820154 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-rq6dq_2e16ce5e-204a-4a42-81b9-0adbcf4c7bd2/istio-proxy/1.log" Sep 11 12:54:44.888782 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.888760 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-ds9t5_6f1b1984-c94f-46c7-bd68-2ad7398b1a19/payload-processing/0.log" Sep 11 12:54:44.924663 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:44.924641 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68f4bb8d9b-qnbg7_29c80efb-0d0b-44ba-b8ec-95d061729afa/router/0.log" Sep 11 12:54:48.818761 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:48.818731 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-tjpvn_8da8d03e-56f4-4451-b35a-4e4a4dfe6f72/kube-storage-version-migrator-operator/1.log" Sep 11 12:54:48.819721 ip-10-0-140-239 kubenswrapper[2575]: I0911 12:54:48.819702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-tjpvn_8da8d03e-56f4-4451-b35a-4e4a4dfe6f72/kube-storage-version-migrator-operator/0.log"